search

pulp-platform / ara

The PULP Ara is a 64-bit Vector Unit, compatible with the RISC-V Vector Extension Version 1.0, working as a coprocessor to CORE-V's CVA6 core
Other
357 stars 127 forks source link

Failed to execute Ideal Dispatcher mode #320

Open Vinay0203 opened 3 months ago

Vinay0203 commented 3 months ago

I am trying to execute fmatmul program in Ideal Dispatcher mode.

The steps i have taken is : 

  1. cd apps
make bin/fmatmul.ideal

    and the output got is : 

    
cd fmatmul && if [ -d script ]; then python3 script/gen_data.py 128 128 128 > data.S ; else touch data.S; fi

sed -i s/"li t0, MSTATUS_FS | MSTATUS_XS$"/"li t0, MSTATUS_FS | MSTATUS_XS | MSTATUS_VS"/ /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/crt.S

git update-index --assume-unchanged /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/crt.S

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -DPREALLOCATE=1 -DSPIKE=1 -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/env -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common -ffunction-sections -fdata-sections -c fmatmul/data.S -o fmatmul/data.S.o.spike

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -DPREALLOCATE=1 -DSPIKE=1 -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/env -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common -ffunction-sections -fdata-sections -c /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/crt.S -o /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/crt.S.o.spike

mkdir -p bin/

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -Iinclude -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -DPREALLOCATE=1 -DSPIKE=1 -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/env -I/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common -ffunction-sections -fdata-sections -o bin/fmatmul.spike fmatmul/kernel/fmatmul.c.o.spike fmatmul/main.c.o.spike fmatmul/data.S.o.spike /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/crt.S.o.spike /home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/syscalls.c.o.spike common/util.c.o.spike -static -nostartfiles -lm -Wl,--gc-sections -nostdlib -T/home/vinay/Documents/ARA_new/ara/apps/riscv-tests/benchmarks/common/test.ld -Wl,--gc-sections -DSPIKE

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/llvm-objdump --mattr=v -D bin/fmatmul.spike > bin/fmatmul.spike.dump

mkdir -p ideal_dispatcher/vtrace ideal_dispatcher/log ideal_dispatcher/temp

echo "run" | /home/vinay/Documents/ARA_new/ara/install/riscv-isa-sim-mod/bin/spike --isa=rv64gcv_zfh --varch="vlen:4096,elen:64" -d bin/fmatmul.spike 2> ideal_dispatcher/temp/fmatmul.temp 1> ideal_dispatcher/log/fmatmul.log

cd ideal_dispatcher && scripts/vtrace.sh temp/fmatmul.temp vtrace/fmatmul.vtrace

The SPIKE log was not clean.

mkdir -p bin/

cp bin/fmatmul.spike bin/fmatmul.ideal


2.

cd hardware make simv app=fmatmul ideal_dispatcher=1

and the output got is :

Makefile:83: "Specified QuestaSim version (questa-2021.2) not found in PATH /home/vinay/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin"

build/verilator/Vara_tb_verilator -l ram,/home/vinay/Documents/ARA_new/ara/apps/bin/fmatmul,elf

bash: line 1: build/verilator/Vara_tb_verilator: No such file or directory

make: *** [Makefile:232: simv] Error 127


3. I am in the up to date main branch and even failed to generate fst waveform for ```fmatmul``` after running this code :

Compile fmatmul for Ara

make -C apps bin/fmatmul

Verilate the design

make -C hardware verilate trace=1

Run fmatmul on Ara through Verilator

make -C hardware simv app=fmatmul trace=1 

and the output got is :

vinay@vinay:~/Documents/ARA_new/ara$ # Compile fmatmul for Ara

make -C apps bin/fmatmul

Verilate the design

make -C hardware verilate trace=1 -B

Run fmatmul on Ara through Verilator

make -C hardware simv app=fmatmul trace=1

make: Entering directory '/home/vinay/Documents/ARA_new/ara/apps'

cd fmatmul && if [ -d script ]; then python3 script/gen_data.py 128 128 128 > data.S ; else touch data.S; fi

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -ffunction-sections -fdata-sections -c fmatmul/data.S -o fmatmul/data.S.o

chmod +x /home/vinay/Documents/ARA_new/ara/apps/common/script/align_sections.sh

rm -f /home/vinay/Documents/ARA_new/ara/apps/common/link.ld && cp /home/vinay/Documents/ARA_new/ara/apps/common/arch.link.ld /home/vinay/Documents/ARA_new/ara/apps/common/link.ld

/home/vinay/Documents/ARA_new/ara/apps/common/script/align_sections.sh 4 /home/vinay/Documents/ARA_new/ara/apps/common/link.ld

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -ffunction-sections -fdata-sections -c fmatmul/main.c -o fmatmul/main.c.o

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -ffunction-sections -fdata-sections -c common/crt0.S -o common/crt0-llvm.S.o

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -ffunction-sections -fdata-sections -c common/printf.c -o common/printf-llvm.c.o

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -ffunction-sections -fdata-sections -c common/string.c -o common/string-llvm.c.o

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -ffunction-sections -fdata-sections -c common/serial.c -o common/serial-llvm.c.o

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -ffunction-sections -fdata-sections -c common/util.c -o common/util-llvm.c.o

mkdir -p bin/

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/clang -Iinclude -march=rv64gcv_zfh_zvfh0p1 -menable-experimental-extensions -mabi=lp64d -mno-relax -fuse-ld=lld -fno-vectorize -mllvm -scalable-vectorization=off -mllvm -riscv-v-vector-bits-min=0 -Xclang -target-feature -Xclang +no-optimized-zero-stride-load -mcmodel=medany -I/home/vinay/Documents/ARA_new/ara/apps/common -std=gnu99 -O3 -ffast-math -fno-common -fno-builtin-printf -DNR_LANES=4 -DVLEN=4096 -Wunused-variable -Wall -Wextra -Wno-unused-command-line-argument -ffunction-sections -fdata-sections -o bin/fmatmul fmatmul/kernel/fmatmul.c.o fmatmul/main.c.o fmatmul/data.S.o common/crt0-llvm.S.o common/printf-llvm.c.o common/string-llvm.c.o common/serial-llvm.c.o common/util-llvm.c.o -static -nostartfiles -lm -Wl,--gc-sections -T/home/vinay/Documents/ARA_new/ara/apps/common/link.ld

ld.lld: warning: ignoring memory region assignment for non-allocatable section '.comment'

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/llvm-objdump --mattr=v -D bin/fmatmul > bin/fmatmul.dump

/home/vinay/Documents/ARA_new/ara/install/riscv-llvm/bin/llvm-strip bin/fmatmul -S --strip-unneeded

rm common/serial-llvm.c.o common/printf-llvm.c.o common/crt0-llvm.S.o common/util-llvm.c.o common/string-llvm.c.o fmatmul/main.c.o

make: Leaving directory '/home/vinay/Documents/ARA_new/ara/apps'

make: Entering directory '/home/vinay/Documents/ARA_new/ara/hardware'

Makefile:83: "Specified QuestaSim version (questa-2021.2) not found in PATH /home/vinay/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin"

mkdir -p build

Bender already exists.

bender 0.27.3 available.

rm -rf build/verilator; mkdir -p build/verilator

/home/vinay/Documents/ARA_new/ara/hardware/../hardware/bender script verilator -t rtl -t cv64a6_imafdcv_sv39 -t tech_cells_generic_include_tc_sram -t tech_cells_generic_include_tc_clk -t ara_test -t cva6_test -t verilator --define NR_LANES=4 --define VLEN=4096 --define ARIANE_ACCELERATOR_PORT=1 --define COMMON_CELLS_ASSERTS_OFF > build/verilator/bender_script_default

/home/vinay/Documents/ARA_new/ara/install/verilator/bin/verilator -f build/verilator/bender_script_default \

-GNrLanes=4 \

-O3 \

-Wno-fatal \

-Wno-PINCONNECTEMPTY \

-Wno-BLKANDNBLK \

-Wno-CASEINCOMPLETE \

-Wno-CMPCONST \

-Wno-LATCH \

-Wno-LITENDIAN \

-Wno-UNOPTFLAT \

-Wno-UNPACKED \

-Wno-UNSIGNED \

-Wno-WIDTH \

-Wno-WIDTHCONCAT \

-Wno-ENUMVALUE \

-Wno-COMBDLY \

tb/verilator/waiver.vlt \

--Mdir build/verilator \

-Itb/dpi \

--compiler clang \

-CFLAGS "-DTOPLEVEL_NAME=ara_tb_verilator" \

-CFLAGS "-DNR_LANES=4" \

-CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_dpi/cpp \

-CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_verilator/cpp \

-CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_simutil_verilator/cpp \

"" \

-LDFLAGS "-lelf" \

"" \

--exe \

/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_dpi/cpp/*.cc \

/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_verilator/cpp/*.cc \

/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_simutil_verilator/cpp/*.cc \

/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/ara_tb.cpp \

--cc \

--trace-fst -Wno-INSECURE \

--top-module ara_tb_verilator && \

cd build/verilator && OBJCACHE='' make -j4 -f Vara_tb_verilator.mk

%Error: Verilator threw signal 9. Suggest trying --debug --gdbbt

%Error: Command Failed ulimit -s unlimited 2>/dev/null; exec /home/vinay/Documents/ARA_new/ara/install/verilator/bin/verilator_bin -f build/verilator/bender_script_default -GNrLanes=4 -O3 -Wno-fatal -Wno-PINCONNECTEMPTY -Wno-BLKANDNBLK -Wno-CASEINCOMPLETE -Wno-CMPCONST -Wno-LATCH -Wno-LITENDIAN -Wno-UNOPTFLAT -Wno-UNPACKED -Wno-UNSIGNED -Wno-WIDTH -Wno-WIDTHCONCAT -Wno-ENUMVALUE -Wno-COMBDLY tb/verilator/waiver.vlt --Mdir build/verilator -Itb/dpi --compiler clang -CFLAGS -DTOPLEVEL_NAME=ara_tb_verilator -CFLAGS -DNR_LANES=4 -CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_dpi/cpp -CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_verilator/cpp -CFLAGS -I/home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_simutil_verilator/cpp -LDFLAGS -lelf --exe /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_dpi/cpp/dpi_memutil.cc /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_dpi/cpp/sv_scoped.cc /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_memutil_verilator/cpp/verilator_memutil.cc /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_simutil_verilator/cpp/verilated_toplevel.cc /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/lowrisc_dv_verilator_simutil_verilator/cpp/verilator_sim_ctrl.cc /home/vinay/Documents/ARA_new/ara/hardware/tb/verilator/ara_tb.cpp --cc --trace-fst -Wno-INSECURE --top-module ara_tb_verilator

make: *** [Makefile:188: build/verilator/Vara_tb_verilator] Error 255

make: Leaving directory '/home/vinay/Documents/ARA_new/ara/hardware'

make: Entering directory '/home/vinay/Documents/ARA_new/ara/hardware'

Makefile:83: "Specified QuestaSim version (questa-2021.2) not found in PATH /home/vinay/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin"

build/verilator/Vara_tb_verilator -t -l ram,/home/vinay/Documents/ARA_new/ara/apps/bin/fmatmul,elf

bash: line 1: build/verilator/Vara_tb_verilator: No such file or directory

make: *** [Makefile:231: simv] Error 127

make: Leaving directory '/home/vinay/Documents/ARA_new/ara/hardware'


4. I am trying to achieve fst file only for the vector unit 'ARA' which does not include signals from CVA6 {or} some extra signals which are dedicated to ARA along with the CVA6 signals
So if we get a trace file in Ideal Dispatcher mode is it the trace file only generated by ARA alone and only that signals? Am i thinking correctly? 
and what is the command modification to get trace file after solving this issue , is it just by giving command as :

make simv app=fmatmul ideal_dispatcher=1 trace=1

mp-17 commented 3 months ago

Hello @Vinay0203, Also other users reported issues with the Verilator traces. I will need some time to check this out since we use Verilator only in the CI with no traces and no ideal-dispatcher mode. If you have a QuestaSim license instead, it should work out of the box

Vinay0203 commented 3 months ago

@mp-17

Please comment on the point 4. illustrated in previous message