search

litex-hub / linux-on-litex-vexriscv

Linux on LiteX-VexRiscv
BSD 2-Clause "Simplified" License
552 stars 174 forks source link

ULX3S builds for multicores and with fpu does not work #267

Closed goran-mahovlic closed 2 years ago

goran-mahovlic commented 2 years ago

I have tried only one core with FPU --with-fpu Multple cores --cpu-count 2 with and wihout FPU and none of that options is working

No output on serial at all

enjoy-digital commented 2 years ago

Hi @goran-mahovlic,

we just had a look with @Dolu1990 and the issue was related to the PBus data-width. This is fixed in simulation with https://github.com/enjoy-digital/litex/commit/967ea12364c3ff307503b5aea17dad701dd5f69c and should also fix the behavior on hardware. Could update LiteX and do a test? Thanks.

goran-mahovlic commented 2 years ago

ULX4M with UM-45F 100MHz 2 cores FPU now starts!

how can I check if it has 2 cores?

https://asciinema.org/a/NeymQ00u6g6llnjXhsdbguSi4

ULX3S 2 core FPU also starts @50MHz as with higher frequency memory fails

Ok I have rebuild for ULX4M 45F and then made a copy of rv32.dtb, and it seams it works stable now

https://asciinema.org/a/NwzfzQqfwEiWYz3BGTBkJQikF

Dolu1990 commented 2 years ago

ULX4M with UM-45F 100MHz 2 cores FPU now starts!

Rawrrrrr 100 Mhz without timing failures :D ?

how can I check if it has 2 cores?

All good on that side ?

goran-mahovlic commented 2 years ago

Rawrrrrr 100 Mhz without timing failures :D ?

Well timings are there to brake :) I do have 1.2V core voltage I think that helps a lot!

I did try on ULX3S with 1.2V core, but it want even start. Will investigate more later. ULX3S could probably work on at least 75MHz but SDRAM memory is failing on that speed

All good on that side ?

Yes, looks all good!

Now I am trying to build some apps, but I am always stuck at building buildroot - I always lose a lot of time to get things running

Dolu1990 commented 2 years ago

I always lose a lot of time to get things running

Same, often it terrify me to even try to bring back to life a board i haven't touched since a while XD

for buildroot, do you have the compilation cache activated and the multithreaded compilation too ? :

Cache : BR2_CCACHE=y BR2_CCACHE_DIR="$(HOME)/.buildroot-ccache-saxon_default"

Parallel compilation of packages : BR2_PER_PACKAGE_DIRECTORIES=y

This can realy help. speeding things up

goran-mahovlic commented 2 years ago

Huh, I do not have those settings, will check

current attempt is to have all needed for buildroot on windows machine inside docker :)

so far it is building nicely ...

building

I would like to have SDL compiles so I can play more easily with framebuffer and maybe even get things compiled with envox studio to start...

goran-mahovlic commented 2 years ago

@Dolu1990 How can I fix this error?

Incorrect selection of kernel headers: expected 5.16.x, got 5.14.x

goran-mahovlic commented 2 years ago

I selected requested version manually, but it failed as before

faild

Dolu1990 commented 2 years ago

header =>

maybe BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_5_14=y

personnaly, i'm using :

git clone https://github.com/buildroot/buildroot.git --branch master cd buildroot git checkout 9ef54b7d

not the upstream buildroot, to avoid hiting new fresh issues XD

I selected requested version manually, but it failed as before

ahhh so you already did the above thing BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_5_14 thing. hmmm So i would say, be sure you make clean after editing / updating BR2_PACKAGE_HOST_LINUX_HEADERS_CUSTOM_5_14

Else, i would say go for buildroot 9ef54b7d

Then, to get SDL and stuff like this, here is a config which has SDL2 + chocolate-doom : https://github.com/enjoy-digital/litex_naxriscv_test/issues/1#issuecomment-1043015332

goran-mahovlic commented 2 years ago

Cool, tnx! Will check with 9ef54b7d - I do have some 56Mb image for SaxonSoc but unfortunately it fails at oms point so I need to rebuild

goran-mahovlic commented 2 years ago

Nop, I am still not able to rebuild

Old image that I have has everything I need, but it stops at some point

fail

This is with running CPU @ 50MHz so that is not a problem

I do have new image, so maybe you have idea what file would generate this error.

I will try to somehow copy paste file from the working rootfs.cpio

Dolu1990 commented 2 years ago

So cause 2 => illegal instruction executed (https://github.com/riscv/riscv-isa-manual/blob/23a1255de3af5d5d6a7c6993dc9f44687446c6d4/src/machine.tex#L2040)

badaddr will refer to that instruction 0x00EF850A, which seems to be a compressed instruction, which is weird.

Could you send the ld-2.32.9000.so file ?

goran-mahovlic commented 2 years ago

I will try to replace that one with "old" one - I just need to find how to pack everything with cpio

https://we.tl/t-1yIbK9jxLA

goran-mahovlic commented 2 years ago

I have replaced that one, but now it stops here

stops

Dolu1990 commented 2 years ago

So, i checked it was compiled with RVC enabled.

00000a00 <_start>: a00: 850a mv a0,sp

I think by default RVC isn't enabled on the VexRiscv litex.

Dolu1990 commented 2 years ago

So, to resume, what are your current buildroot settings, and what are your current litex command lines ?

goran-mahovlic commented 2 years ago

This image is some old SaxonSoc complete and big image that I was thinking of reusing

I am not able to rebuild as I constantly end with errors like this

https://github.com/litex-hub/linux-on-litex-vexriscv/issues/267#issuecomment-1079047484

it may have something with WSL as I am building with Windows

Dolu1990 commented 2 years ago

Ahhh so, 9ef54b7d + disabling RVC combo may help

About the kernel, you are using the one provided by litex right ?

goran-mahovlic commented 2 years ago

Ahhh so, 9ef54b7d + disabling RVC combo may help

I am currently @9ef54b7d but how can I disable RVC combo?

About the kernel, you are using the one provided by litex right ?

I have tried many options but this one is currently set

options

goran-mahovlic commented 2 years ago

If this is RVC it is not set :(

RVC

Dolu1990 commented 2 years ago

be carefull, BR2_RISCV_ISA_CUSTOM_RVC is not set does not mean it is turned off, it may mean it has its default value, which may be enabled by default

Also one thing which may help, is make xconfig this will instead run a GUI, in which you have ctrl + f <3

goran-mahovlic commented 2 years ago

@Dolu1990 I have installed everything from beginning in VirtualBox and now compilation goes to end

But now it seams that rootfs is not containing all needed libraries to run doom

doom

But when I check cpio file I do find them inside of /usr/lib folder

usr_lib

When system boots there are not inside - so I am probably missing something?

Dolu1990 commented 2 years ago

Ahhh that's weird, i never had that issue.

Can you check LD_LIBRARY_PATH ? and the existance of the file from the running system perspective ?

goran-mahovlic commented 2 years ago

I am rebuilding everything with litex_vexriscv_defconfig + added SDL and doom as I am also missing /dev/mmc with saxon config

Will check LD_LIBRARY_PATH once build finish

goran-mahovlic commented 2 years ago

I have mmc back, and doom, but still missing those files

echo $LD_LIBRARY_PATH does not return anything

path

where would I need to set it?

Dolu1990 commented 2 years ago

So i would say, check that the file are still there on the target runtime, depending how you flashed the sdcard they may be missing (especialy the link) about LD_LIBRARY_PATH , it being not set isn't a issue in itself, but as a desesparate attempt i would for instance set it on the target runtime with a export LD_LIBRARY_PATH=/usr/lib

goran-mahovlic commented 2 years ago

I have copied usr/lib to SD card and did export but it seams new system is not recognizing framebuffer as video

usr

Dolu1990 commented 2 years ago

do you have /dev/fb0 ? If yes, it mean that x11 isn't running

goran-mahovlic commented 2 years ago

do you have /dev/fb0 ?

yes

If yes, it mean that x11 isn't running

tnx, I was missing that one in config - need to add more packets to get it ...

goran-mahovlic commented 2 years ago

I have now manage to get Xorg that was missing, but then I was missing some more SDL files and added few more things to config

And now I have lost MMC as device, and do not know how to get that one back?

@enjoy-digital

Is there a way to bring up hw manually, or do yo know what option is bringing MMC back

MMC

I have checked module is listed in modules.builtin, but it seams kernel dir is empty

modules_builtin kernel

Maybe it has to do something with some defined size of file system?

As all /usr is also missing

https://github.com/litex-hub/linux-on-litex-vexriscv/issues/267#issuecomment-1079706924

I was also able to compile https://github.com/eez-open/modular-psu-firmware/tree/1.8.alpha2 that I want to check - it would be cool if I can get that one running as we could easily create measuring instrument, and if we manage to get it run on Sigilent that would be a big deal

With some options I can get back to mmc support, and with mounted /usr/lib on SDCARD I can get this

failed

That is same error I get when I try to run doom

Failed_2

DirectFB/core/system: No system found!

Ok solved that one by copy directfb to /usr/lib

now it asks for fusion

Dolu1990 commented 2 years ago

@goran-mahovlic How did you loaded the file system ? via a cpio file loaded as a ramfs ?

Ok solved that one by copy directfb to /usr/lib

This kind of maniuplation are realy weird, so far all the buildroot images i got were realy "compile, flash, go". No tricky manipulation involved.

also in your run, i can see that chocolate-doom is trying to run on directFB instead of X11, did you exported DISPLAY=:0 ?

but then I was missing some more SDL files and added few more things to config

So i remember that because of some FUTEX issue in the libc, sdl2 was disabled in buildroot for RISC-V if that the case in your config too, then i find the change that has to be done to make it work.

goran-mahovlic commented 2 years ago

@goran-mahovlic How did you loaded the file system ? via a cpio file loaded as a ramfs ?

I am using this as a startup https://github.com/litex-hub/linux-on-litex-vexriscv#-generating-the-linux-binaries-optional and it creates cpio file that completely loads itself into ram - if I use your sample then I lose mmc SD card device and cpio for some reason does not contain /usr/lib that contains SDL libraries and I cannot start anything

This kind of maniuplation are realy weird, so far all the buildroot images i got were realy "compile, flash, go". No tricky manipulation involved.

Yes, I remember that was working nicely SaxonSoc and there complete system was on SD card, but I do not know what changed or how can I change

also in your run, i can see that chocolate-doom is trying to run on directFB instead of X11, did you exported DISPLAY=:0 ?

I have tried, but it did not work - I think now it does not work as I have re-build everything with threaded enabled as that is what modular-psu-firmware - so that may be a reason it now ask for fusion - I do not know how to tell SDL app not to use fusion so I have enabled fusion - and it creates lib and udev but it does not seams to load kernel module needed

doom

So i remember that because of some FUTEX issue in the libc, sdl2 was disabled in buildroot for RISC-V if that the case in your config too, then i find the change that has to be done to make it work.

Will check FUTEX

Dolu1990 commented 2 years ago

cpio for some reason does not contain /usr/lib that contains SDL libraries and I cannot start anything

Ahhh then, about loading in ram, you will likely have to extends the dts region allocated for it. In the DTS is defined by : 

        chosen {
            bootargs = "console=liteuart earlycon=liteuart,0xf0001000 rootwait root=/dev/ram0";
            linux,initrd-start = <0x41000000>;   // <------
            linux,initrd-end   = <0x41800000>;  // <------ only 8MB allocated for the cpio
        };

I'm surprised, isn't the ulx3s limited to 64 MB ? so you are testing on the ULX4M right, with all its sexy DDR3 ?

Will check FUTEX

Here is are the workaround, totaly unsafe <3:

https://github.com/SpinalHDL/buildroot-spinal-saxon/blob/main/external.mk#L17 https://github.com/SpinalHDL/buildroot-spinal-saxon/blob/main/external.mk#L18

goran-mahovlic commented 2 years ago

I'm surprised, isn't the ulx3s limited to 64 MB ? so you are testing on the ULX4M right, with all its sexy DDR3 ?

yes,yes it is for UL4M and I currently have 512Mb DDR3 - where is that DTS file?

Here is are the workaround, totaly unsafe <3:

I already added that :)

Dolu1990 commented 2 years ago

where is that DTS file

It is autogenerated in the build/xxx/ folder.

To translate it into dtb => dtc -O dtb -o xxx.dtb xxx.dts

goran-mahovlic commented 2 years ago

Once I remove comment this line I can get doom to build and MMC working - if that is not commented I do not get MMC at boot

https://github.com/goran-mahovlic/linux-on-litex-vexriscv/blob/master/buildroot/board/litex_vexriscv/linux.config#L132

It is autogenerated in the build/xxx/ folder.

Hmm I still do not understand how to get that change, but yes that is left to solve to get complete disk in ram I found dts files all over the place, and I am not finding any dts file. Is that maybe in OpenSBI as I use that for bootloader?

oh, that is step pack while building bitstream - I will check now

goran-mahovlic commented 2 years ago

Woohoo, it works, sorry on all trouble with issues :)

works

Dolu1990 commented 2 years ago

Ahhh nice, gg ^^

About the dts, it is generated by https://github.com/enjoy-digital/litex/blob/master/litex/tools/litex_json2dts_linux.py#L75, so you may have a arguement in command line to feed the proper size.

I was about to point you to this patch for better doom frame rate : https://github.com/SpinalHDL/buildroot-spinal-saxon/blob/main/patches/chocolate-doom/0000-vexriscv.patch

But it seems you already have it in XD One thing, the speed of "real world software" as doom, is very sensitive to the cache size of the CPU.

goran-mahovlic commented 2 years ago

Cool! Yes I do have patch

But I now need to rebuild multi tread version to get eez working - maybe now when I have bigger ram disk fusion will start

goran-mahovlic commented 2 years ago

Got it!

eez-simulator

Colors and size are in the mess, it also take some time to start

Dolu1990 commented 2 years ago

Rawrrrrr ^^ About the boot time, how much i$, d$, cores, Mhz do you have ? What are the litex arguements you are using ?

goran-mahovlic commented 2 years ago

I have some problems on boot now as once I edited dtb I got some error so it is much slower If I fix that one boot time could be around 20sec if those are seconds...

[   14.665039]  mmcblk0: p1
[   31.669841] Initramfs unpacking failed: invalid magic at start of compressed archive
[   42.065129] Freeing initrd memory: 105848K

I am currently way of edge ECP5 should work - but as I have core voltage at 1.2V things are still working But as it is edgy when I change something in core and make new bitstream often I something fails

Currently working setup

100MHz 2 Cores with FPU with 2K of L2 and MMC

--cpu-count=2 --with-wishbone-memory --with-fpu
CPU:            VexRiscv SMP-LINUX @ 100MHz
BUS:            WISHBONE 32-bit @ 4GiB
CSR:            32-bit data
ROM:            64KiB
SRAM:           6KiB
L2:             2KiB
SDRAM:          524288KiB 16-bit @ 400MT/s (CL-6 CWL-5)

--========== Initialization ============--
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Read leveling:
  m0, b00: |01110000| delays: 02+-01
  m0, b01: |00000000| delays: -
  m0, b02: |00000000| delays: -
  m0, b03: |00000000| delays: -
  best: m0, b00 delays: 02+-01
  m1, b00: |01110000| delays: 02+-01
  m1, b01: |00000000| delays: -
  m1, b02: |00000000| delays: -
  m1, b03: |00000000| delays: -
  best: m1, b00 delays: 02+-01
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
  Write: 0x40000000-0x40200000 2.0MiB     
   Read: 0x40000000-0x40200000 2.0MiB     
Memtest OK
Memspeed at 0x40000000 (Sequential, 2.0MiB)...
  Write speed: 25.3MiB/s
   Read speed: 34.6MiB/s
Dolu1990 commented 2 years ago

--cpu-count=2 --with-wishbone-memory --with-fpu

Hooo one thing, i think @enjoy-digital did a fix on the data mask of the DDR, and now you may not need the --with-wishbone-memory anymore, which would boost performance quite a bit i think.

It should be visible in the memspeed test. Just be sure to have the last versions of Litex / litedram ^^

goran-mahovlic commented 2 years ago

Hooo one thing, i think @enjoy-digital did a fix on the data mask of the DDR, and now you may not need the --with-wishbone-memory anymore, which would boost performance quite a bit i think.

cool, will need to try out

It should be visible in the memspeed test. Just be sure to have the last versions of Litex / litedram ^^

Will rebuild everything and check if it will work

As now I have build options for eez I have tried to build for raspberry pi and it is working quite well - but it would be amazing if we could run that on Sigilent - maybe just the screen with options, without frame that shows power supply

https://youtu.be/FQ8vjHNZNlY

You can check how simulator is working here: https://www.envox.eu/bb3-simulator/index.html

And maybe with few tweaks in GUI we could probably get even more on Osciloscope :)

goran-mahovlic commented 2 years ago

--cpu-count=2 --with-wishbone-memory --with-fpu

Hooo one thing, i think @enjoy-digital did a fix on the data mask of the DDR, and now you may not need the --with-wishbone-memory anymore, which would boost performance quite a bit i think.

It should be visible in the memspeed test. Just be sure to have the last versions of Litex / litedram ^^

Once I remove --with-wishbone-memory I cannot even get one core to finish boot, but by boot time I can see that it could give X2 speed

Also setting setting cache to 0 on my board does not boot pass memory test

https://github.com/litex-hub/linux-on-litex-vexriscv/issues/253

goran-mahovlic commented 2 years ago

Also adding more L2 cache makes memtest to fail - I have tried one 100MHz CPU with 64K L2 --with-wishbone-memory

Now I am trying to build 2 cores 50Mhz with 64K L2 --with-wishbone-memory

Dolu1990 commented 2 years ago

I cannot even get one core to finish boot,

Which boot ? the linux one ?

This may be due to timing violating right ?

enjoy-digital commented 2 years ago

The initial issue has been solved, we can probably close this. Great to see Doom on the ULX3S :)

jaycbhsn commented 2 years ago

I have copied usr/lib to SD card and did export but it seams new system is not recognizing framebuffer as video

usr

I am running into the same issue with the same library.

libSDL2-2.0.so.0

Could you explain what exactly you placed in /usr/lib or what you did to resolve this?

Dolu1990 commented 2 years ago

@jaycbhsn Can you show your buildroot defconfig ?