Toolchain issues

[chchenl]

1. The dowload_xxx.sh files seem not-working(using ubuntu 23.x or 24.x)

2. In vicuna RTL sim, if using built-in downloaded toolchain(version 22.x), when the below command is issued(as prescribed in Sim/Vicuna/Readme file): "cmake -DSIMULATOR=Vicuna -DRISCV_GCC_PREFIX=$(pwd)/../Toolchain/rv32imv/ -DENABLE_INTG_TESTS=ON -DTOOLCHAIN=GCC -DUSE_VEXT=ON .."

an error message saying that it cannot find 'multilib' required for the architecture rv32im_zvxxxx_zlxxx in the toolchain rv32imv (I cannot capture the screen right now. Probably I'll upload it sometime later.) Or rather, a more detailed error message is: "cannot find suitable multilib set for -march=rv32im_zve32x_zvl32b'/' -mabi=ilp32.....", then this cmake command failed and terminated.

This raises another issue: is the built-in(downloaded) toolchain buit with '--enable-multilib' option enabled? From the readme file, I guess the recommended approach for building one's own toolchain does not enable this option.

Or if this issue is related to versions of gcc or any other utilities? I use gcc-13.2.

Thanks for any info.

--vincent

[PhilippvK]

I am sorry for the inconvenience…

I currently provide various GCC (13/14) & LLVM (15-19) builds for Ubuntu 20.04 & 22.04. Unfortunately I ran out of the 50GB auf space I can use on the could service provider. To also upload 24.04 variants of the tools I would need ~5GB more which is currently difficult to free up. There is a likelihood that the 22.04 binaries also work on 24.04, so you could give it a try by hardcoding the Ubuntu version detected by the script.

Regarding multilib support: Actually I have uploaded multilib toolschains to the Sync&Share yesterday. Feel free to give them a try, but it is highly likely that they will not work together with LLVM because RISC-V multilib support is basically broken since forever. This is the reason why we have to maintain many non-multilib builds instead of only one multilib-GCC. I would love to eliminate this issue, but guess it’s up to the RISC-V & LLVM community to deal with that.

Some further notes:

• rv32im is actually not a valid -march=, as v also implies fd_zve32f_zve64x_zve64f_zve64d which are all unsupported by zve32x targeted by Vicuna
• I would recommend to use rv32im_zve32x_zicsr_zifencei (Newer GCC versions need the Zicsr & Zifencei as well)
• In case you don’t know: Vicuna only supports a subset of the Zve32x spec, hence programs compiled with GCC/LLVM (especially with autovectorization enabled) will likely not work out of the box.
• Further there are bugs in vicuna which need to be worked around
• We actually have a more-complete version of vicuna which might be worth a look (after @ParkerJones567 git the time to open-source it)
• The Scripts to compile & run muRISCV on vicuna you can find in this repository are probably outdated and will eventually not work.

If you need guidance in setting up muRISCV-NN and Vicuna, please feel free to reach out to us!

[PhilippvK]

Here you can browse for the available Toolchains: https://syncandshare.lrz.de/getlink/fiWBtDLWz17RBc1Yd4VDW7/

[chchenl]

Hi, Thanks for providing detailed info. 　

I actually tried rv32im_zve32x_zicsr_zifencei and an error message appeared as below. Could you comment on how this could be solved? Thanks in advance.

===========================================

vincent@vincent-debug:~/Proj/muriscv-nn/build$ cmake -DSIMULATOR=Vicuna -DRISCV_GCC_PREFIX=$(pwd)/../Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32 -DENABLE_INTG_TESTS=ON -DTOOLCHAIN=GCC -DUSE_VEXT=ON .. -- CMake Version 3.27.4 -- The C compiler identification is GNU 13.2.0 -- The CXX compiler identification is GNU 13.2.0 -- The ASM compiler identification is GNU -- Found assembler: /home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc -- Detecting C compiler ABI info -- Detecting C compiler ABI info - failed -- Check for working C compiler: /home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc -- Check for working C compiler: /home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc - broken CMake Error at /usr/share/cmake-3.27/Modules/CMakeTestCCompiler.cmake:67 (message): The C compiler

"/home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc"

is not able to compile a simple test program.

It fails with the following output:

Change Dir: '/home/vincent/Proj/muriscv-nn/build/CMakeFiles/CMakeScratch/TryCompile-v99uC9'

Run Build Command(s): /usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_65845/fast
/usr/bin/gmake  -f CMakeFiles/cmTC_65845.dir/build.make CMakeFiles/cmTC_65845.dir/build
gmake[1]: Entering directory '/home/vincent/Proj/muriscv-nn/build/CMakeFiles/CMakeScratch/TryCompile-v99uC9'
Building C object CMakeFiles/cmTC_65845.dir/testCCompiler.c.obj
/home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc   -march=rv32imzve32x -mabi=ilp32 -mcmodel=medany  -o CMakeFiles/cmTC_65845.dir/testCCompiler.c.obj -c /home/vincent/Proj/muriscv-nn/build/CMakeFiles/CMakeScratch/TryCompile-v99uC9/testCCompiler.c
Linking C executable cmTC_65845
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_65845.dir/link.txt --verbose=1
/home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc  -march=rv32imzve32x -mabi=ilp32 -mcmodel=medany   -march=rv32imzve32x -mabi=ilp32 -mcmodel=medany  CMakeFiles/cmTC_65845.dir/testCCompiler.c.obj -o cmTC_65845 
riscv32-unknown-elf-gcc: fatal error: Cannot find suitable multilib set for '-march=rv32im_zve32x_zvl32b'/'-mabi=ilp32'
compilation terminated.
gmake[1]: *** [CMakeFiles/cmTC_65845.dir/build.make:99: cmTC_65845] Error 1
gmake[1]: Leaving directory '/home/vincent/Proj/muriscv-nn/build/CMakeFiles/CMakeScratch/TryCompile-v99uC9'
gmake: *** [Makefile:127: cmTC_65845/fast] Error 2

CMake will not be able to correctly generate this project. Call Stack (most recent call first): CMakeLists.txt:97 (project)

-- Configuring incomplete, errors occurred!

[PhilippvK]

Could you comment on how this could be solved?

@chchenl Of course. Since the TC was built with zicsr_zifencei, the extensions also need to show up in the RISCV_ARCH. This was not an issue in earlier GCC versions because there the two extensions where automatically implied and not error was thrown if the lookup failed.

This should fix your build issue:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 17300ed..93c72f7 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -44,7 +44,7 @@ option(USE_PEXT "Use the RISC-V P Packed Extension" OFF)
 # - rv32imzve32x / ilp32 -> integer only and embedded vector support (GCC only supports rv32imv!)
 if("${SIMULATOR}" STREQUAL "Vicuna") # Vicuna only supports a reduced set of extensions
   if(USE_VEXT)
-    set(RISCV_ARCH "rv32imzve32x" CACHE STRING "march argument to the compiler, using the vector extension")
+    set(RISCV_ARCH "rv32im_zicsr_zifencei_zve32x" CACHE STRING "march argument to the compiler, using the vector extension")
   elseif(USE_PEXT)
     message(FATAL_ERROR "Vicuna does not support packed instructions.")
   else()

[chchenl]

Hi,

I modified the CMakeLists.txt accordingly, but it seems the error message remains the same.

There seems to be 3 suspicious messages

Is this the reason why the cmake command failed? Thanks for info.

(BTW, don't know why the font size here gets so large--sorry about that. --Fix now, the font size is much smaller.)

================

cmake -DSIMULATOR=Vicuna -DRISCV_GCC_PREFIX=$(pwd)/../Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32 -DENABLE_INTG_TESTS=ON -DTOOLCHAIN=GCC -DUSE_VEXT=ON .. -- CMake Version 3.27.4 -- The C compiler identification is GNU 13.2.0 -- The CXX compiler identification is GNU 13.2.0 -- The ASM compiler identification is GNU -- Found assembler: /home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc -- Detecting C compiler ABI info -- Detecting C compiler ABI info - failed -- Check for working C compiler: /home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc -- Check for working C compiler: /home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc - broken CMake Error at /usr/share/cmake-3.27/Modules/CMakeTestCCompiler.cmake:67 (message): The C compiler

"/home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc"

is not able to compile a simple test program.

It fails with the following output:

Change Dir: '/home/vincent/Proj/muriscv-nn/build/CMakeFiles/CMakeScratch/TryCompile-tYjS6q'

Run Build Command(s): /usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_b7209/fast
/usr/bin/gmake  -f CMakeFiles/cmTC_b7209.dir/build.make CMakeFiles/cmTC_b7209.dir/build
gmake[1]: Entering directory '/home/vincent/Proj/muriscv-nn/build/CMakeFiles/CMakeScratch/TryCompile-tYjS6q'
Building C object CMakeFiles/cmTC_b7209.dir/testCCompiler.c.obj
/home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc   -march=rv32imzve32x -mabi=ilp32 -mcmodel=medany  -o CMakeFiles/cmTC_b7209.dir/testCCompiler.c.obj -c /home/vincent/Proj/muriscv-nn/build/CMakeFiles/CMakeScratch/TryCompile-tYjS6q/testCCompiler.c
Linking C executable cmTC_b7209
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_b7209.dir/link.txt --verbose=1
/home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc  -march=rv32imzve32x -mabi=ilp32 -mcmodel=medany   -march=rv32imzve32x -mabi=ilp32 -mcmodel=medany  CMakeFiles/cmTC_b7209.dir/testCCompiler.c.obj -o cmTC_b7209 
**_riscv32-unknown-elf-gcc: fatal error: Cannot find suitable multilib set for '-march=rv32im_zve32x_zvl32b'/'-mabi=ilp32'_**
compilation terminated.
gmake[1]: *** [CMakeFiles/cmTC_b7209.dir/build.make:99: cmTC_b7209] Error 1
gmake[1]: Leaving directory '/home/vincent/Proj/muriscv-nn/build/CMakeFiles/CMakeScratch/TryCompile-tYjS6q'
gmake: *** [Makefile:127: cmTC_b7209/fast] Error 2

CMake will not be able to correctly generate this project. Call Stack (most recent call first): CMakeLists.txt:98 (project)

[chchenl]

Another message near the end...:

/home/vincent/Proj/muriscv-nn/Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32/bin/riscv32-unknown-elf-gcc -march=rv32imzve32x -mabi=ilp32 -mcmodel=medany -march=rv32imzve32x -mabi=ilp32 -mcmodel=medany CMakeFiles/cmTC_b7209.dir/testCCompiler.c.obj -o cmTC_b7209 _riscv32-unknown-elf-gcc: fatal error: Cannot find suitable multilib set for '-march=rv32im_zve32xzvl32b'/'-mabi=ilp32' compilation terminated.

[PhilippvK]

@chchenl That’s odd. I tried the fix and it worked for me as expected.

Here is some advice for further investigation:

• Maybe the old RISCV_ARCH variable ist being cached by CMake. Please remove the build directory before re-running cmake.
• Add a MESSAGE(STATUS "RISCV_ARCH=${RISCV_ARCH}") at the end of the top-level CMakeLists.txt to verify if the variable is correctly set
• Make sure that you use the rv32im_zicsr_zifencei_zve32x non-multilib toolchain. I will try to provide instructions for the multilib gcc later!

[chchenl]

Hi, You are right. The issue seems related to the cached variable under cmake mechanism. With a new 'build' directory, the original errors are gone.

Here is a new message, which seems to indicate I skipped some steps(downloading something?)--but I cannot tell if I skipped something from the readme file. I suspect it resuled from no-running Integration/tflm/download_tflm.sh along the way. But isn't this supposed to run automatically or invoked by some other (intermediate) script?

I tried to manually issue 'download_tflm.sh', but the directory structure generated does not coincide with the directory structure the 'error message' expected, therefore you may have to manually adjust its structures. I guess this is not a 'standard' approach, though.

cmake -DSIMULATOR=Vicuna -DRISCV_GCC_PREFIX=$(pwd)/../Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32 -DENABLE_INTG_TESTS=ON -DTOOLCHAIN=GCC -DUSE_VEXT=ON .. -- CMake Version 3.27.4 -- Populating unity -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /home/vincent/Proj/muriscv-nn/build/_deps/unity-subbuild [ 11%] Performing update step for 'unity-populate' [ 22%] No patch step for 'unity-populate' [ 33%] No configure step for 'unity-populate' [ 44%] No build step for 'unity-populate' [ 55%] No install step for 'unity-populate' [ 66%] No test step for 'unity-populate' [ 77%] Completed 'unity-populate' [100%] Built target unity-populate -- Successfully added test_muriscv_nn_avgpool_s8 -- Successfully added test_muriscv_nn_avgpool_s16 -- Successfully added test_muriscv_nn_convolve_1x1_s8_fast -- Successfully added test_muriscv_nn_convolve_s8 -- Successfully added test_muriscv_nn_convolve_s16 -- Successfully added test_muriscv_nn_depthwise_conv_3x3_s8 -- Successfully added test_muriscv_nn_depthwise_conv_fast_s16 -- Successfully added test_muriscv_nn_depthwise_conv_s8 -- Successfully added test_muriscv_nn_depthwise_conv_s8_opt -- Successfully added test_muriscv_nn_depthwise_conv_s16 -- Successfully added test_muriscv_nn_elementwise_add_s8 -- Successfully added test_muriscv_nn_elementwise_add_s16 -- Successfully added test_muriscv_nn_elementwise_mul_s8 -- Successfully added test_muriscv_nn_elementwise_mul_s16 -- Successfully added test_muriscv_nn_fully_connected_s8 -- Successfully added test_muriscv_nn_fully_connected_s16 -- Successfully added test_muriscv_nn_maxpool_s8 -- Successfully added test_muriscv_nn_maxpool_s16 -- Successfully added test_muriscv_nn_relu_q7 -- Successfully added test_muriscv_nn_relu6_s8 -- Successfully added test_muriscv_nn_softmax_s8 -- Successfully added test_muriscv_nn_softmax_s8_s16 -- Successfully added test_muriscv_nn_softmax_s16 -- Successfully added test_muriscv_nn_svdf_s8 -- Successfully added test_muriscv_nn_svdf_state_s16_s8 -- Successfully added test_muriscv_nn_lstm_unidirectional_s8 -- Successfully added test_muriscv_nn_lstm_unidirectional_s16 -- Successfully added aww_tflm -- Successfully added ic_tflm -- Successfully added toy_tflm -- Successfully added vww_tflm -- Successfully added aww_tvm -- Successfully added ic_tvm -- Successfully added toy_tvm -- Successfully added vww_tvm -- RISCV_ARCH=rv32im_zicsr_zifencei_zve32x -- Configuring done (0.3s) CMake Error at Integration/tflm/CMakeLists.txt:406 (target_sources): Cannot find source file:

/home/vincent/Proj/muriscv-nn/Integration/tflm/tensorflow/lite/micro/kernels/cmsis_nn/fully_connected.cc

Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .ccm .cxxm .c++m .h .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc

CMake Error at Integration/tflm/CMakeLists.txt:19 (add_library): No SOURCES given to target: tflm

[chchenl]

Hi PhilippvK,

Another idea occurred to me. Or rather, if my goal is to make sure the toolchain as well as the base database is properly installed and good to go, is the issued 'cmake' command one of the last few verification steps to that goal? I surmised I was following the right reame file(under https://github.com/tum-ei-eda/muriscv-nn/tree/master/Sim/Vicuna) to reach that goal.

Or if I am on the wrong path to that goal, could you show me a proper path/sequence to that goal? Thanks.

[PhilippvK]

Hi, You are right. The issue seems related to the cached variable under cmake mechanism. With a new 'build' directory, the original errors are gone.

Here is a new message, which seems to indicate I skipped some steps(downloading something?)--but I cannot tell if I skipped something from the readme file. I suspect it resuled from no-running Integration/tflm/download_tflm.sh along the way. But isn't this supposed to run automatically or invoked by some other (intermediate) script?

I tried to manually issue 'download_tflm.sh', but the directory structure generated does not coincide with the directory structure the 'error message' expected, therefore you may have to manually adjust its structures. I guess this is not a 'standard' approach, though.

cmake -DSIMULATOR=Vicuna -DRISCV_GCC_PREFIX=$(pwd)/../Toolchain/rv32im_zicsr_zifencei_zve32x_ilp32 -DENABLE_INTG_TESTS=ON -DTOOLCHAIN=GCC -DUSE_VEXT=ON .. -- CMake Version 3.27.4 -- Populating unity -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /home/vincent/Proj/muriscv-nn/build/_deps/unity-subbuild [ 11%] Performing update step for 'unity-populate' [ 22%] No patch step for 'unity-populate' [ 33%] No configure step for 'unity-populate' [ 44%] No build step for 'unity-populate' [ 55%] No install step for 'unity-populate' [ 66%] No test step for 'unity-populate' [ 77%] Completed 'unity-populate' [100%] Built target unity-populate -- Successfully added test_muriscv_nn_avgpool_s8 -- Successfully added test_muriscv_nn_avgpool_s16 -- Successfully added test_muriscv_nn_convolve_1x1_s8_fast -- Successfully added test_muriscv_nn_convolve_s8 -- Successfully added test_muriscv_nn_convolve_s16 -- Successfully added test_muriscv_nn_depthwise_conv_3x3_s8 -- Successfully added test_muriscv_nn_depthwise_conv_fast_s16 -- Successfully added test_muriscv_nn_depthwise_conv_s8 -- Successfully added test_muriscv_nn_depthwise_conv_s8_opt -- Successfully added test_muriscv_nn_depthwise_conv_s16 -- Successfully added test_muriscv_nn_elementwise_add_s8 -- Successfully added test_muriscv_nn_elementwise_add_s16 -- Successfully added test_muriscv_nn_elementwise_mul_s8 -- Successfully added test_muriscv_nn_elementwise_mul_s16 -- Successfully added test_muriscv_nn_fully_connected_s8 -- Successfully added test_muriscv_nn_fully_connected_s16 -- Successfully added test_muriscv_nn_maxpool_s8 -- Successfully added test_muriscv_nn_maxpool_s16 -- Successfully added test_muriscv_nn_relu_q7 -- Successfully added test_muriscv_nn_relu6_s8 -- Successfully added test_muriscv_nn_softmax_s8 -- Successfully added test_muriscv_nn_softmax_s8_s16 -- Successfully added test_muriscv_nn_softmax_s16 -- Successfully added test_muriscv_nn_svdf_s8 -- Successfully added test_muriscv_nn_svdf_state_s16_s8 -- Successfully added test_muriscv_nn_lstm_unidirectional_s8 -- Successfully added test_muriscv_nn_lstm_unidirectional_s16 -- Successfully added aww_tflm -- Successfully added ic_tflm -- Successfully added toy_tflm -- Successfully added vww_tflm -- Successfully added aww_tvm -- Successfully added ic_tvm -- Successfully added toy_tvm -- Successfully added vww_tvm -- RISCV_ARCH=rv32im_zicsr_zifencei_zve32x -- Configuring done (0.3s) CMake Error at Integration/tflm/CMakeLists.txt:406 (target_sources): Cannot find source file:

/home/vincent/Proj/muriscv-nn/Integration/tflm/tensorflow/lite/micro/kernels/cmsis_nn/fully_connected.cc

Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .ccm .cxxm .c++m .h .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc

CMake Error at Integration/tflm/CMakeLists.txt:19 (add_library): No SOURCES given to target: tflm

For reference an setting up and running the integration tests please refer to the CI scripts: https://github.com/tum-ei-eda/muriscv-nn/blob/master/.github/workflows/integration_tests.yml

Instead of invoking cmake manually, you can use the run_tflm_integration_tests.sh which should setup the directory structure.

Due to the various limitations/bugs in the upstream version of Vicuna ist it unlikely that the integration tests will work out-of-the box…

[PhilippvK]

Hi PhilippvK,

Another idea occurred to me. Or rather, if my goal is to make sure the toolchain as well as the base database is properly installed and good to go, is the issued 'cmake' command one of the last few verification steps to that goal? I surmised I was following the right reame file(under https://github.com/tum-ei-eda/muriscv-nn/tree/master/Sim/Vicuna) to reach that goal.

Or if I am on the wrong path to that goal, could you show me a proper path/sequence to that goal? Thanks.

The mentioned in the last post, the best „documentation/example“ on how to setup the flow is to look at the ci scripts which should be working 100% this week. It currently just uses Spike as Somulator, but we can look into integrating Vicuna there as well.