All `targets/RISC-V/spike32.py` tests failing

TommyMurphyTM1234 commented 2 weeks ago

I am using the following:

riscv-gnu-toolchain built from commit 763c862
riscv-openocd built from commit c6bb902629924eb66aae2a08c0ab8654261c9d71
riscv-tests at commit b0eb63a

and when I run the Spike32 tests:

./gdbserver.py ./targets/RISC-V/spike32.py --isolate --print-failures --remotelogfile-enable --server_cmd "openocd -d3" 2>&1 | tee test-spike32.log

all tests fail:

/gdbserver.py ./targets/RISC-V/spike32.py --isolate --print-failures --remotelogfile-enable --server_cmd "openocd -d3" 2>&1 | tee test-spike32.log
Using $misa from hart definition: 0x4034112d
PRNG seed for spike32 is generated automatically
PRNG seed for spike32 is 1719915165
[CeaseStepiTest] Starting > logs/20240702-111245-spike32-CeaseStepiTest.log
[CeaseStepiTest] not_applicable in 0.00s
[CheckMisa] Starting > logs/20240702-111245-spike32-CheckMisa.log
[CheckMisa] exception in 3.57s
Test: CheckMisa
Target: spike32
---------------------------------[ Compile ]----------------------------------
+ riscv64-unknown-elf-gcc -g programs/checksum.c programs/tiny-malloc.c programs/infinite_loop.S -DDEFINE_MALLOC -DDEFINE_FREE -DCLINT=33554432 programs/entry.S programs/init.c -DNHARTS=1 -I ../env -T targets/RISC-V/spike32.lds nostartfiles -mcmodel=medany -DXLEN=32 -o /tmp/spike32_checksum-4034112d_h3qffwsl -march=rv32imafdcv_zicsr -mabi=ilp32
-------------------------[ /tmp/spike-bti84wl5.log ]--------------------------
+ spike -p1 --isa RV32IMAFDCV_Zvl128b_Zve64d --dm-auth --dmi-rti 4 --dm-no-abstract-fpr --dm-no-halt-groups -m0x10100000:0x10000000 --rbb-port 0 /tmp/spike32_checksum-4034112d_h3qffwsl
error: bad --isa option 'RV32IMAFDCV_Zvl128b_Zve64d'. unsupported extension: zvl128b

--------------------------------[ Traceback ]---------------------------------
Traceback (most recent call last):
  File "/home/user/Downloads/openocd-test/riscv-tests/debug/testlib.py", line 1379, in run
    self.classSetup()
  File "/home/user/Downloads/openocd-test/riscv-tests/debug/testlib.py", line 1435, in classSetup
    BaseTest.classSetup(self)
  File "/home/user/Downloads/openocd-test/riscv-tests/debug/testlib.py", line 1342, in classSetup
    self.target_process = self.target.create()
  File "/home/user/Downloads/openocd-test/riscv-tests/debug/./targets/RISC-V/spike32.py", line 24, in create
    return testlib.Spike(self, isa="RV32IMAFDCV", dmi_rti=4,
  File "/home/user/Downloads/openocd-test/riscv-tests/debug/testlib.py", line 120, in __init__
    raise TestLibError("Didn't get spike message about bitbang "
testlib.TestLibError: Didn't get spike message about bitbang connection
-------------------------------[ End of logs ]--------------------------------
Result: exception
Logfile: logs/20240702-111245-spike32-CheckMisa.log
Reproduce: ./gdbserver.py ./targets/RISC-V/spike32.py CheckMisa
Time elapsed: 3.57s
[CustomRegisterTest] Starting > logs/20240702-111249-spike32-CustomRegisterTest.log
[CustomRegisterTest] exception in 3.59s
Test: CustomRegisterTest
Target: spike32
---------------------------------[ Compile ]----------------------------------
+ riscv64-unknown-elf-gcc -g programs/checksum.c programs/tiny-malloc.c programs/infinite_loop.S -DDEFINE_MALLOC -DDEFINE_FREE -DCLINT=33554432 programs/entry.S programs/init.c -DNHARTS=1 -I ../env -T targets/RISC-V/spike32.lds nostartfiles -mcmodel=medany -DXLEN=32 -o /tmp/spike32_checksum-4034112d_dgmrsm6l -march=rv32imafdcv_zicsr -mabi=ilp32
-------------------------[ /tmp/spike-lhs71q6e.log ]--------------------------
+ spike -p1 --isa RV32IMAFDCV_Zvl128b_Zve64d --dm-auth --dmi-rti 4 --dm-no-abstract-fpr --dm-no-halt-groups -m0x10100000:0x10000000 --rbb-port 0 /tmp/spike32_checksum-4034112d_dgmrsm6l
error: bad --isa option 'RV32IMAFDCV_Zvl128b_Zve64d'. unsupported extension: zvl128b

--------------------------------[ Traceback ]---------------------------------
Traceback (most recent call last):
  File "/home/user/Downloads/openocd-test/riscv-tests/debug/testlib.py", line 1379, in run
    self.classSetup()
  File "/home/user/Downloads/openocd-test/riscv-tests/debug/testlib.py", line 1435, in classSetup
    BaseTest.classSetup(self)
  File "/home/user/Downloads/openocd-test/riscv-tests/debug/testlib.py", line 1342, in classSetup
    self.target_process = self.target.create()
  File "/home/user/Downloads/openocd-test/riscv-tests/debug/./targets/RISC-V/spike32.py", line 24, in create
    return testlib.Spike(self, isa="RV32IMAFDCV", dmi_rti=4,
  File "/home/user/Downloads/openocd-test/riscv-tests/debug/testlib.py", line 120, in __init__
    raise TestLibError("Didn't get spike message about bitbang "
testlib.TestLibError: Didn't get spike message about bitbang connection
-------------------------------[ End of logs ]--------------------------------
Result: exception
Logfile: logs/20240702-111249-spike32-CustomRegisterTest.log
Reproduce: ./gdbserver.py ./targets/RISC-V/spike32.py CustomRegisterTest
Time elapsed: 3.59s

...

:::::::::::::::::::::::::::[ ran 73 tests in 222s ]:::::::::::::::::::::::::::
12 tests returned not_applicable
61 tests returned exception
   CheckMisa > logs/20240702-111245-spike32-CheckMisa.log
   CustomRegisterTest > logs/20240702-111249-spike32-CustomRegisterTest.log
   DebugBreakpoint > logs/20240702-111253-spike32-DebugBreakpoint.log
   DebugChangeString > logs/20240702-111256-spike32-DebugChangeString.log
   DebugCompareSections > logs/20240702-111300-spike32-DebugCompareSections.log
   DebugExit > logs/20240702-111304-spike32-DebugExit.log
   DebugFunctionCall > logs/20240702-111307-spike32-DebugFunctionCall.log
   DebugSymbols > logs/20240702-111311-spike32-DebugSymbols.log
   DebugTurbostep > logs/20240702-111314-spike32-DebugTurbostep.log
   DisconnectTest > logs/20240702-111318-spike32-DisconnectTest.log
   DownloadTest > logs/20240702-111321-spike32-DownloadTest.log
   EbreakTest > logs/20240702-111325-spike32-EbreakTest.log
   EtriggerTest > logs/20240702-111329-spike32-EtriggerTest.log
   FreeRtosTest > logs/20240702-111332-spike32-FreeRtosTest.log
   Hwbp1 > logs/20240702-111336-spike32-Hwbp1.log
   Hwbp2 > logs/20240702-111340-spike32-Hwbp2.log
   HwbpManual > logs/20240702-111343-spike32-HwbpManual.log
   IcountTest > logs/20240702-111347-spike32-IcountTest.log
   InfoTest > logs/20240702-111351-spike32-InfoTest.log
   InstantChangePc > logs/20240702-111354-spike32-InstantChangePc.log
   InstantHaltTest > logs/20240702-111358-spike32-InstantHaltTest.log
   InterruptTest > logs/20240702-111402-spike32-InterruptTest.log
   ItriggerTest > logs/20240702-111405-spike32-ItriggerTest.log
   JumpHbreak > logs/20240702-111409-spike32-JumpHbreak.log
   MemTest16 > logs/20240702-111413-spike32-MemTest16.log
   MemTest32 > logs/20240702-111416-spike32-MemTest32.log
   MemTest64 > logs/20240702-111420-spike32-MemTest64.log
   MemTest8 > logs/20240702-111423-spike32-MemTest8.log
   MemTestBlock0 > logs/20240702-111427-spike32-MemTestBlock0.log
   MemTestBlock1 > logs/20240702-111431-spike32-MemTestBlock1.log
   MemTestBlock2 > logs/20240702-111434-spike32-MemTestBlock2.log
   MemTestReadInvalid > logs/20240702-111438-spike32-MemTestReadInvalid.log
   PrivChange > logs/20240702-111442-spike32-PrivChange.log
   PrivRw > logs/20240702-111445-spike32-PrivRw.log
   ProgramHwWatchpoint > logs/20240702-111449-spike32-ProgramHwWatchpoint.log
   ProgramSwWatchpoint > logs/20240702-111453-spike32-ProgramSwWatchpoint.log
   Registers > logs/20240702-111456-spike32-Registers.log
   RepeatReadTest > logs/20240702-111500-spike32-RepeatReadTest.log
   Semihosting > logs/20240702-111503-spike32-Semihosting.log
   SemihostingFileio > logs/20240702-111507-spike32-SemihostingFileio.log
   SimpleF18Test > logs/20240702-111511-spike32-SimpleF18Test.log
   SimpleNoExistTest > logs/20240702-111514-spike32-SimpleNoExistTest.log
   SimpleS0Test > logs/20240702-111518-spike32-SimpleS0Test.log
   SimpleS1Test > logs/20240702-111522-spike32-SimpleS1Test.log
   SimpleT0Test > logs/20240702-111525-spike32-SimpleT0Test.log
   SimpleT1Test > logs/20240702-111529-spike32-SimpleT1Test.log
   SimpleV13Test > logs/20240702-111532-spike32-SimpleV13Test.log
   StepTest > logs/20240702-111536-spike32-StepTest.log
   Sv32Test > logs/20240702-111540-spike32-Sv32Test.log
   TooManyHwbp > logs/20240702-111544-spike32-TooManyHwbp.log
   TriggerDmode > logs/20240702-111547-spike32-TriggerDmode.log
   TriggerExecuteInstant > logs/20240702-111551-spike32-TriggerExecuteInstant.log
   TriggerLoadAddressInstant > logs/20240702-111554-spike32-TriggerLoadAddressInstant.log
   TriggerStoreAddressInstant > logs/20240702-111558-spike32-TriggerStoreAddressInstant.log
   UnavailableCycleTest > logs/20240702-111602-spike32-UnavailableCycleTest.log
   UnavailableHaltedTest > logs/20240702-111605-spike32-UnavailableHaltedTest.log
   UnavailableRunTest > logs/20240702-111609-spike32-UnavailableRunTest.log
   UserInterrupt > logs/20240702-111613-spike32-UserInterrupt.log
   VectorTest > logs/20240702-111616-spike32-VectorTest.log
   WriteCsrs > logs/20240702-111620-spike32-WriteCsrs.log
   WriteGprs > logs/20240702-111624-spike32-WriteGprs.log

Summary log: spike32.log

The problem seems to be this:

+ spike -p1 --isa RV32IMAFDCV_Zvl128b_Zve64d --dm-auth --dmi-rti 4 --dm-no-abstract-fpr --dm-no-halt-groups -m0x10100000:0x10000000 --rbb-port 0 /tmp/spike32_checksum-4034112d_h3qffwsl
error: bad --isa option 'RV32IMAFDCV_Zvl128b_Zve64d'. unsupported extension: zvl128b

But...

I don't know what triggered this?
I don't know why the spike32.py uses a different ISA specification (including V) to other targets and why there isn't more consistency here?
I don't know why the CI runs don't seem to be affected (yet?)?

aap-sc commented 2 weeks ago

I don't know what triggered this?

This is like related to changes in upstream spike. If you look at the current changelog there is lot's of changes related to how RVV configuration is handled.

I don't know why the spike32.py uses a different ISA specification (including V) to other targets and why there isn't more consistency here?

I think this we should discuss this in https://github.com/riscv-software-src/riscv-tests/issues/568

I don't know why the [CI runs](https://github.com/riscv-collab/riscv-openocd/actions) don't seem to be affected (yet?)?

The current TOT in spike is 1b1a33376 . Which correspond to the latest pipeline we have in OpenOCD. It seems that you spike version is deprecated.

aap-sc commented 2 weeks ago

@TommyMurphyTM1234

This is like related to changes in upstream spike. If you look at the current changelog there is lot's of changes related to how RVV configuration is handled.

In addition to my point above you can also see here: https://github.com/riscv-software-src/riscv-tests/commit/e06a435c1e545def71e833031356372f0828f165 . So yes, this is changes in upstream spike

TommyMurphyTM1234 commented 2 weeks ago

Thanks @aap-sc.

The current TOT in spike is 1b1a33376 . Which correspond to the latest pipeline we have in OpenOCD. It seems that you spike version is deprecated.

I just used the version that the riscv-gnu-toolchain repo uses as a submodule. So maybe that needs to be bumped over there?

aap-sc commented 2 weeks ago

I just used the version that the riscv-gnu-toolchain repo uses as a submodule. So maybe that needs to be bumped over there?

Have no opinion on the matter since we don't use riscv-gnu-toolchain :(

TommyMurphyTM1234 commented 1 week ago

As mentioned above, riscv-gnu-toolchain currently uses https://github.com/riscv-software-src/riscv-isa-sim/commit/20a2b6d05a562236a32ca3ce1642a2bdbbd907a0 for its Spike submodule. This is about three months old and predates the changes related to V extension handling.

I did a local build of Spike at the current top of tree commit - https://github.com/riscv-software-src/riscv-isa-sim/commit/f7d0dba6012d49cb75b2f175a31a85cf4b2db3b3 - and with this version I no longer have the aforementioned spike32.py test failures.

So the issue is indeed that riscv-gnu-toolchain uses a Spike that is not up to date with the requirements of the OpenOCD tests.

I will see if I can safely bump riscv-gnu-tolchain's Spike submodule to use the latest Spike commit. I need to run the tests suites for the bare-metal/Newlib and Linux toolchains just to check that there is no adverse impact from doing this.

TommyMurphyTM1234 commented 1 week ago

Addressed by this PR:

https://github.com/riscv-collab/riscv-gnu-toolchain/pull/1509

which is also discussed in this issue:

https://github.com/riscv-collab/riscv-gnu-toolchain/issues/1507

riscv-collab / riscv-openocd

All `targets/RISC-V/spike32.py` tests failing #1098