litex_sim: VexRiscV-SMP / Zephyr supported?

[pottendo]

hi,

I successfully ran the bare metal demo: on a litex_sim vexriscv 'cpu'. Inspired by @Dolu1990 *), I've tried to run some Zephyr applications on litex_sim.

However, building and running Zephyr application won't lift-off on the litex_sim using a vexriscv CPU. I tried with building Zephyr's application with west build --pristine -b qemu_riscv32 . (stock Zephyr) and with a modified device-tree (see attachment) to meet certain addresses (like ram and uart).

Before bothering too long, first simple questions:

• can a zephyr applications work in principle with litex_sim on a vexriscv CPU (even aon vexriscv_smp)?
• litex_sim won't run the bare metal demo with a vexriscv_smp CPU (at least in my setup). What is there reason here?

thanx, pottendo

PS: I have a litex based FPGA board running successfully Zephyr applications with VexRiscV CPU and its -smp cousin. Linux works as well (on -smp of course). See here and here - for basic usage, adjustment to the specific devices of the board in the devicetree were necessary on the Zephyr side.

*) in some thread around enhancing VexRiscV-SMP's FPU support.

PPS: thanks for litex and all the related RiscV ecosystem I'm using... great stuff!

zephyr-dts.tar.gz

[Dolu1990]

Hi,

can a zephyr applications work in principle with litex_sim on a vexriscv CPU (even aon vexriscv_smp)?

Never tried, but it works on the board right ? that's weird it doesn't works in litex_sim. Maybe some peripheral mapping is moving around when in simulation ?

Else, i can take a look at a wave + binaries + elf file if you sent them to me, or i have a way to reproduce.

litex_sim won't run the bare metal demo with a vexriscv_smp CPU (at least in my setup). What is there reason here?

Likely, the demo isn't made to handle multiple CPU poping in its init vector. The boot process from the bios is : CPU0 run the bios, while other CPU wait in a while loop, then CPU0 wake up all the others CPU to run the payload, then CPU0 jump to the payload. If the payload isn't made to handle to additional CPU, things will turn bad :)

Regards Charles

PPS: thanks for litex and all the related RiscV ecosystem I'm using... great stuff!

<3

[pottendo]

hi, thanks for the response. Find attached the binaries and some more files from the Zephyr build, which may help. This is the version where I've adjusted the device-tree.

And yes, on my target the program (Zephyr apps in general) work.

Further background: I was doing this because I experience problems in Zephyr on vexriscv-smp with 'CONFIG_SMP' enabled on Zephyr on my target hardware. Here also the system won't lift-off. I've tracked that down to 'spinlock' never unlocking , as far as I can see in Zephyr (Riscv arch). But I wanted to see if this can be reproduced on litex_sim to make life easier for anyone who can look into it (i.e. you experts ;-) ) Note that Linux/SMP on the same bitstream on my FPGA works as expected (i.e. 2 cores and a significant speedup when using multithreaded apps) - so this must be Zephyr vs. my board specific. I can share more on this, but it would be off-topic in this issue. zephyr-hello-workd.zip

thanx, pottendo

[Dolu1990]

Ahhh, i think it maybe due to the fact that atomic memory operation are not suported on IO access, only on the main memory (litedram) That's probably it :) Don't know if you can set litedram stuff in sim as main memory ?

[Dolu1990]

looking at your elf file, seems like it is already in litedram mapping XD Hmmm

[Dolu1990]

What shell command you did to run the simulation ?

[pottendo]

What shell command you did to run the simulation ? $ litex_sim --cpu-type=vexriscv --integrated-main-ram-size=$((65536*2*4)) --ram-init ~/zephyr-scratch/zephyr/samples/hello_world/build/zephyr/zephyr.bin

[Dolu1990]

Hmm and which command had zephyr working on hardware ?

[pottendo]

Hmm and which command had zephyr working on hardware ?

I launch via litex_term , basically the same cmd-line (don't have it handy now), without the memory-size stuff. From memory this looks like this: litex_term --kernel .../zephyr.bin /dev/ttyACM0 This brings up the BIOS and downloads the binary via USB and lifts-off. It also works to boot via a 'boot.json' file from the SDCard of the board.

Of course the Zephyr binary is built using my orangecart board: west build -b orangecart <path> instead of ...-b qemu-riscv32 .... So it's a different binary. thx, pottendo

[Dolu1990]

Hoo my bad, i'm mixing different issues threads XD I'm looking at it :)

[Dolu1990]

first issue is that the binary is compiled with RVC enabled, can see the cpu goes to the first RVC instruction before crashing :

Then it got a instruction decoding exception : Because vexriscv most of the time has the PMP implementation disabled to save area. and zephyr tries to use it without catching potential non implementation.

Can you disable the PMP usage from zephyr ?

i used the following for the sim : litex_sim --cpu-type=vexriscv_smp --integrated-main-ram-size=$((6553624)) --ram-init zephyr.bin --trace --trace-fst --with-rvc

[pottendo]

Hi, I disabled by 'CONFIG_RISCV_PMP=n' in the respective Zephyr config. Not sure if this is fully effective. Using 'vexriscv_smp' now lifts-off with showing one character... so maybe just the wrong baudrate?

I think we're close... thumbs up, pottendo

Update: tried to align baudraute with `--uart-baudrate=115200' on litex_sim and the devicetree accordingly

&uart0 {
    status = "okay";
    current-speed = <115200>;
};

did change by now seeing two characters...

PS: enable the trace results in ModuleNotFoundError: No module named 'vcd.gtkw' what and how to install this?

[hardiselg]

enable the trace results in ModuleNotFoundError: No module named 'vcd.gtkw' what and how to install this?

I have the same issue, I have installed gtkwave, pyvcd and pip install vcd Verilog_VCD. Nothing works.

[Dolu1990]

did change by now seeing two characters...

Yes likely baudrate then :D

ModuleNotFoundError: No module named 'vcd.gtkw'

--trace-fst => It isn't VCD anymore, instead, you can open build/sim/gateware/sim.fst with gtkwave ^^

[hardiselg]

--trace-fst => It isn't VCD anymore, instead, you can open build/sim/gateware/sim.fst with gtkwave ^^

This worked, thanks!

As I understand --trace is deprecated?

[pottendo]

did change by now seeing two characters...

Yes likely baudrate then :D

as said in my update, aligning baudrates didn't help to see a correct output. Need to see the Zephyr driver...

ModuleNotFoundError: No module named 'vcd.gtkw'

--trace-fst => It isn't VCD anymore, instead, you can open build/sim/gateware/sim.fst with gtkwave ^^ not exactly in my setup :-(

$ gtkwave build/sim/gateware/sim.fst
GTKWave Analyzer v3.3.104 (w)1999-2020 BSI

Segmentation fault (core dumped)

gtkwave without the .lst starts. (attached the .fst file)
[fst.zip](https://github.com/enjoy-digital/litex/files/10792358/fst.zip)

[Dolu1990]

Ahh one thing, is that the FST file is compressed, and so, Verilator only dump data in it once there is enough data. So, you may have to let the simulation run longer, until the file size take a MB.

Was it the issue ?

[pottendo]

hi, good news:

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Executing booted program at 0x40000000

--============= Liftoff! ===============--
*** Booting Zephyr OS build 8732a08ed13d ***
Hello World! orangecart_smp

I've changed my orangecart config as well to CONFIG_RISCV_PMP=n, aligned the baudrate and, voila, it works! So the driver (ns16550) selected in Zephyr's qemu boardconfig won't work but driver `liteuart' does.

So now to my 'CONFIG_SMP=y' problem... stay tuned (maybe see you in another issue, ;-) ).

Ad FST: 'running longer' doesn't work in this simple scenario... it stops too quickly after lift-off. Will check with different scenarios later...

so long, pottendo

[pottendo]

OK, not totally surprising:

$litex_sim --cpu-type=vexriscv_smp --with-rvc --cpu-count=2 --trace-fst
...
[clocker] loaded
[serial2tcp] loaded (0x55ec78e08ef0)
[spdeeprom] loaded (addr = 0x0)
[serial2console] loaded (0x55ec78e08ef0)
[ethernet] loaded (0x55ec78e08ef0)
[gmii_ethernet] loaded (0x55ec78e08ef0)
[xgmii_ethernet] loaded (0x55ec78e08ef0)
[clocker] sys_clk: freq_hz=1000000, phase_deg=0

locks up here.

This still fits to this thread - let me know if vexriscv_smp with --cpu-count=2 is supposed to be supported by litex_sim. If not, let us know and you may close this issue then.

thanks for the support, pottendo

PS: Tracing generates files, but I don't know how to use them:

-rw-rw-r-- 1 pottendo pottendo    330 Feb 21 11:54 sim.fst
-rw-rw-r-- 1 pottendo pottendo 327680 Feb 21 11:54 sim.fst.hier

neither are accepted by gtkwave.

[Dolu1990]

--trace-fst alone isn't enough you also need --trace :) (sim.fst of 330 bytes mean there is no data in it.)

$litex_sim --cpu-type=vexriscv_smp --with-rvc --cpu-count=2 --trace-fst

So it is a wishbone size missmatch, adding --wishbone-force-32b fixes it

[hardiselg]

adding --trace generates module not found error

litex_sim --with-sdram --sdram-init=demo/demo.bin --cpu-type=cv32e40p --cpu-variant=full --trace --trace-fst ... Traceback (most recent call last): File "/usr/bin/litex_sim", line 33, in sys.exit(load_entry_point('litex', 'console_scripts', 'litex_sim')()) File "/home/hartz/.local/lib/python3.10/site-packages/litex/tools/litex_sim.py", line 494, in main builder.build( File "/home/hartz/.local/lib/python3.10/site-packages/litex/soc/integration/builder.py", line 364, in build vns = self.soc.build(build_dir=self.gateware_dir, kwargs) File "/home/hartz/.local/lib/python3.10/site-packages/litex/soc/integration/soc.py", line 1287, in build return self.platform.build(self, *args, *kwargs) File "/home/hartz/.local/lib/python3.10/site-packages/litex/build/sim/platform.py", line 54, in build return self.toolchain.build(self, args, kwargs) File "/home/hartz/.local/lib/python3.10/site-packages/litex/build/sim/verilator.py", line 247, in build pre_run_callback(v_output.ns) File "/home/hartz/.local/lib/python3.10/site-packages/litex/tools/litex_sim.py", line 491, in pre_run_callback generate_gtkw_savefile(builder, vns, args.trace_fst) File "/home/hartz/.local/lib/python3.10/site-packages/litex/tools/litex_sim.py", line 332, in generate_gtkw_savefile with gtkw.GTKWSave(vns, savefile=savefile, dumpfile=dumpfile) as save: File "/home/hartz/.local/lib/python3.10/site-packages/litex/build/sim/gtkwave.py", line 76, in enter from vcd.gtkw import GTKWSave ModuleNotFoundError: No module named 'vcd.gtkw

[Dolu1990]

@hardiselg --trace isn't depecated, that's just that vcd wave are big and slow. so adding --trace-fst speed up things. About the error you got, that's weird i do not have it, Do you use a old version of litex ?

[pottendo]

--trace-fst alone isn't enough you also need --trace :)

Same as @hardisleg just reported. I've cloned litex (and even did litex_setup.py --update) last week, so I think I'm quite recent.

Another good news: --wishboned-force-32b launches with --cpu-count=2 and even runs Zephyr's hello world testprogram (without Zephyr SMP). So now I'm there where I originally started - I think I can reproduce the issue with Zephyr applications using 'CONFIG_SMP=y' on a vexriscv_smp CPU. I'll look into it now - I hope I get tracing working, then it'll be more efficient to debug. I think for this I should open another issue; it seems unrelated to litex_sim. @Dolu1990, what do you think? thanx, pottendo

[hardiselg]

Updated litex and everything else today. Without --trace the same command works fine. I can run my code on the CV32E40P just fine.

Tho I am running an older pip 22.3.1

[Dolu1990]

@pottendo Right, i think we can create another issue for the zephyr SMP part :) @hardiselg Seems like your issue would need it's own issue thread too :)

[pottendo]

@hardiselg, pip3 install PyVCD did the trick for me...

[hardiselg]

@pottendo Unfortunately not working for me.

Will create a new thread.

EDIT: Had to reinstall the PyVCD module, then it started working. Thank you for the help!