Allowing multiple threads (HARTs) to run the same binary (bare metal)

[cassebas]

Hi Eugene, I have been trying to run a bare metal riscv program on a dual core rocket64, e.g. the rocket64 with 2 cores and shared L2 cache. I can run the program I have compiled by putting the binary on the SD card. I see only output from the first HART.

When I connect to the FPGA with xsdb, I can attach to the second HART and let the second HART execute the code. But I haven't figured out how to run both HARTs simultaneously.

I have tried other riscv programs as well, e.g. a benchmark from the riscv-tests github repository. Some of these benchmarks have the 'mt-' letters before the name, which I believe stands for multithreaded. The code has a threadentry function, for which I would like them to be called by both HARTs. No such luck however.

Do you know how to make both HARTs active simultaneously, in a bare metal environment? Thank you!

[eugene-tarassov]

In short, you need to implement handler for ecall instruction (aka environment or system call). The secondary cores are waiting for interrupt, then execute ecall. You trigger inter-core interrupt from the first core, then catch the secondary core in the ecall trap handler. In the handler you create and start secondary thread.

[cassebas]

Thank you for your quick response. If I understand you correctly, part of what you say is already present in the bootrom. The HARTs start in the .hang section, and when the mhartid is zero, then the thread jumps to _start_bootrom. Otherwise (other HARTs), the corresponding thread will continue in the _hartx_loop and execute a wfi.

So I should add a part where the first core triggers an interprocessor interrupt. There is already an ecall instruction after the wfi, so I implement the trap handler such that the other cores can jump to this new trap handler. In the trap handler I can create all necessary stuff for the secondary core and jump to main. Is this what you mean?

[eugene-tarassov]

Yes. In the bootrom, the HART 0 reads and starts an application from SD card, all other HARTs are waiting for interrupt. In your bare metal application, you setup a trap for ecall, trigger inter-HART interrupt, and the second HART starts to execute your code.

[cassebas]

Hi Eugene The bootrom has code that places the address 0x80000000 in the mtvec register. The actual helloworld bare metal program starts at 0x80000008. In the dissassembly of helloworld (see below), the first 8 bytes are empty because the main.lds file keeps these bytes empty for the trap vector (according to the comment).

Should these 8 bytes be actually filled with a small trap table? With one address, where this one address is the interrupt service routine? Or is the number 8 bytes arbitrarily chosen and could it be larger?

Thanks again!

Disassembly of section .text:

0000000080000000 <_start-0x8>:
        ...

0000000080000008 <_start>:
    80000008:   0810011b                addiw   sp,zero,129
    8000000c:   01811113                slli    sp,sp,0x18
    80000010:   494000ef                jal     ra,800004a4 <main>

0000000080000014 <_hang>:
    80000014:   10500073                wfi
    80000018:   ffdff06f                j       80000014 <_hang>

[eugene-tarassov]

All traps into machine mode cause the pc to be set to the address in the MTVEC register, or 0x80000000 in this case. You need to replace the empty area at 0x8000000 with your trap handler. Yes, the number 8 bytes arbitrarily chosen, your handler can be larger.

Also, have a look at RISCV Bare Metal. It looks like a good example of exception handling in RISC-V bare metal.

[cassebas]

Thanks for your advice, my problem has been solved! I made the trap entry at 0x80000000 to catch the 2nd hart. The 1st hart writes a '1' to the memory mapped MSIP register in the CLINT controller, with which I can wake the 2nd hart.