OpenMP

[Kendidi]

OpenMP

I wonder does the ESP toolchain support OpenMP?

I want to offload tasks from the the main core to a cluster of cores. Any recommendations on using it, adding it, or via any other method? Thanks a lot in advance.

[paulmnt]

Hi @Kendidi I believe the default configurations of the RISC-V toolchain for Ariane and SPARC toolchain for Leon3 do not enable OpenMP, but I don't see any ESP-specific reason why that should not be possible.

You need to modify the configuration passed to Buildroot for RISC-V or SPARC such that Gcc can parse and compile OpenMP pragmas. For sure you need to add the following: BR2_GCC_ENABLE_OPENMP=y

There might be more flags to add and it might be better to create the default configuration for Buildroot, then enter the temporary build folder (e.g. for RISC-V /tmp/_riscv_build/buildroot) and use make menuconfig to search for OpenMP and add all necessary dependencies automatically. Once you complete the configuration, simply run make install from from the build folder to compile again.

Could you please define what you mean by main core and cluster of cores? Are you integrating your own cluster of custom processors as an accelerator tile?

[Kendidi]

Thank you @paulmnt for your advice! I will look into that.

Yes, we are considering to integrate a cluster of RISC-V cores to the FPGA, similar to what described at PULP platform. Therefore I am wondering how the main ESP core (Runner Linux) can use OpenMP to delegate tasks to the cores in the cluster.

[paulmnt]

This sounds very interesting! Thank you for sharing the information and please let us know if you need additional support.

[Kendidi]

Will do. Thanks again @paulmnt! Appreciate it!

[Kendidi]

Hi @paulmnt,

After I added BR2_GCC_ENABLE_OPENMP=y to esp/utils/toolchain/riscv_buildroot_defconfig, I ran esp/utils/toolchain/build_riscv_toolchain.sh to rebuild root file system but I could not find folder, /tmp/_riscv_build/buildroot or so to run `make menuconfig'. What have I missed? Thanks.

[paulmnt]

/tmp gets cleared when you reboot, but I assume you did not reboot the machine. The folder should be there, unless you changed the script. If you open esp/utils/toolchain/build_riscv_toolchain.sh you will see the path pointing to /tmp.

Note that in this case you need to rebuild the entire toolchain, because you're changing GCC, so you cannot skip the first steps.

[Kendidi]

Yes, I have shut down the system last night. Since bootup, I have rebuilt root file system and ran make linux.

Okay I will rebuilt the toolchain as well. Thanks.

[Kendidi]

I see the directory now. I have tried make menuconfig and also checked out.config but couldn't find anything related to OpenMP is there except # BR2_PACKAGE_OPENMPI is not set.

[paulmnt]

OPENMPI is probably something else. Did you try compiling with OpenMP pragmas after adding BR2_GCC_ENABLE_OPENMP=y. Perhaps you don't need to add anything else.

[Kendidi]

No, I haven't try compiling with OpenMP pragmas yet. That's OK. I will research more on it. Thanks!

On the other hand, I cross compiled a simple application with /opt/riscv/bin/riscv64-unknown-linux-gnu-gcc, put the executable into esp/soft/ariane/sysroot/root/, ran make linux at esp/socs/xilinx-vcu128-xcvu37p. I then boot up the FPGA with the newly created linux.bin. But I can't find the application I just created in /root/ directory. Is there something I may have missed?

[paulmnt]

If you are using the latest ESP commit, then sysroot ifor VCU128 is at the following path: esp/socs/xilinx-vcu128-xcvu37p/soft-build/ariane/sysroot

You may still have esp/socs/xilinx-vcu128-xcvu37p/sysroot from an old build, but that folder is not used anymore.

[Kendidi]

Ooh. I see. Thank you @paulmnt !! It works now.

[paulmnt]

We restructured several paths before the release to make things more flexible. The new soft-build hierarchy allows you to build software for Ariane, Ibex and Leon3 at the same time and avoids the need to run a make distclean when you switch from one core to the other in the GUI.

[Kendidi]

Nice!

If I want to reserve memory that will not be used by Linux OS, but can be access by applications via mmap(). Where is the preferred location to do it?

[paulmnt]

We already do that for our accelerators because Ariane (and the RISC-V ISA it implements) has no flush instruction for the L1 data cache. Therefore, when we want to operate in LLC-coherent or non-coherent mode, we must bypass Ariane's L1 and the way to do so is by reserving some memory that the OS will not use for regular software execution.

On the FPGA build we have OS memory from 0x8000 0000 to 0x9FFF FFFF, Ethernet reserved memory from 0xA000 0000 to 0xA01F FFFF and accelerators' memory from 0xA020 0000 to 0xBFFF FFFF, of which 0xB000 0000 to 0xBFFF FFFF gets reserved for third-party accelerators when they are present.

I think that using the third-party accelerator memory would be the best starting point for your application. Please check the NVDLA tutorial, which shows how the third-party accelerator integration works. I believe this is exactly what you want to do to integrate the cluster of cores.

Please note that the reserved memory region is specified in the device tree esp/socs/xilinx-vcu128-xcvu37p/socgen/esp/riscv.dts. The device tree is generated when you run make esp-xconfig or make esp-config and if you add a third-party accelerator it will create the reserved memory region that you need.

There are more details that you may need related to cache coherence and to the third-party accelerators, but I think for now you can work without instantiating the ESP cache hierarchy in the GUI. This way your cluster should operate correctly when you use the memory region 0xB000 0000 to 0xBFFF FFFF and call mmap() to expose that area to software.

If you need more memory, you can use the entire accelerator memory region 0xA020 0000 to 0xBFFF FFFF and simply remove from sysroot/etc/init.d all the scripts that load the ESP memory allocator and drivers: S64esp and S65drivers. This way you are free to mmap() the whole region and it won't conflict with the ESP memory allocator.

[Kendidi]

Thank you @paulmnt very much for the detailed info.! I was looking at the reserved-memory node in esp/socs/xilinx-vcu128-xcvu37p/socgen/esp/riscv.dts but wasn't sure where to start editing.

I will check out the NVDLA tutorial. Thanks!