Regression Test issue when building SimEng : getrandom syscall

elfmath commented 8 months ago

Goals

As a end-user of SimEng, I wish to build SimEng and pass regression tests before installing it. So that I can trust my installation

Issue

"getrandom syscall" tests failed. Tests failed for AArch64, RISCV with the following Simulation-Mode : emulation, in order and outoforder

From LastTest.log trace file :

[  FAILED  ] AArch64/Syscall.getrandom/emulation, where GetParam() = (0, "{}")
[  FAILED  ] AArch64/Syscall.getrandom/inorder, where GetParam() = (1, "{}")
[  FAILED  ] AArch64/Syscall.getrandom/outoforder, where GetParam() = (2, "{L1-Data-Memory: {Interface-Type: Fixed}}")

[...]

[  FAILED  ] RISCV/Syscall.getrandom/emulation, where GetParam() = (0, "{}")
[  FAILED  ] RISCV/Syscall.getrandom/inorder, where GetParam() = (1, "{}")
[  FAILED  ] RISCV/Syscall.getrandom/outoforder, where GetParam() = (2, "{L1-Data-Memory: {Interface-Type: Fixed}}")

When looking in the file we have this failure for all 6 tests above :

..../SimEng/test/regression/aarch64/Syscall.cc:1574: Failure
Expected equality of these values:
  allMatch
    Which is: true
  false
[  FAILED  ] AArch64/Syscall.getrandom/outoforder, where GetParam() = (2, "{L1-Data-Memory: {Interface-Type: Fixed}}") (10 ms)

Reproduction

Try to build SimEng with :

OS : Linux CentOS 7 (kernel 3.10)
State of the git : tag 0.9.6
cmake version 3.21.4
gcc 7.5.0
LLVM precompiled version 14.0.5

Commands to compile and run test before installation :

mkdir -p $SIMENG_BUILD_DIR
cmake -B $SIMENG_BUILD_DIR -S $SIMENG_SRC_DIR -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=$SIMENG_HOME -DSIMENG_ENABLE_TESTS=ON -DSIMENG_USE_EXTERNAL_LLVM=ON -DLLVM_DIR=$LLVM_DIR
cmake --build $SIMENG_BUILD_DIR -j{nproc}
cmake --build $SIMENG_BUILD_DIR --target test

jj16791 commented 8 months ago

Hi @elfmath.

Thank you for raising this issue and detailing the problem well. What OS are you running on? I've just run the same commands and compiler on a SUSE Linux system and all the tests pass. I'm thinking there may be an OS-Compiler combination which causes the above issues you're seeing.

elfmath commented 8 months ago

Hello @jj16791,

I'm working on a Linux CentOS 7 machine to build and execute SimEng. You raised a good point on the OS/compiler combination. I tried to execute tests on Ubuntu 22.04, and, like you, the tests passed.

So I tried to debug it and I pointed out the issue for my environment. "getrandom" test is calling a system call (number 278). This syscall is emulated using srand with clock() as a seed. According to the manual clock() should return the number of CPU cycles since the begining of the program. So the result should has a granularity of 1 (like on Ubuntu). However for an unknown reason yet, the implementation of clock on my machine has a granularity of 10 000.

That beeing said, the seed for srand is the same for the two consecutive calls to the syscall so the pseudo random generated by rand() are also the same. That is why the test failed. One solution is to put srand in the constructor of ExceptionHandler with a constante seed. For example srand(100). However, I think this fix is not good for the simulator since we have a deterministic way of generate random number, with the same sequences.

NB : get random syscall is located at : src/lib/arch/aarch64/ExceptionHandler.cc::608 (for SimEng 0.9.6)

jj16791 commented 8 months ago

Hi @elfmath,

Thank you for the OS information and the debugging work done. We'll have a discussion internally to decide what the next best steps are and provide any relevant updates here.

UoB-HPC / SimEng