riscv-software-src / riscof

BSD 3-Clause "New" or "Revised" License
63 stars 40 forks source link

Exit code is success after an error #102

Open Barabas5532 opened 10 months ago

Barabas5532 commented 10 months ago

I'm using the sail_cSim plugin as reference, and it seems like some errors reported by the plugin cause an exit with a success code. This is quite dangerous for CI usage, as a CI system will believe riscof test ran successfully.

Current behaviour:

When a test fails to execute, riscof exits with a success code.

Expected behaviour:

Riscof exits with a success code only when all of the tests pass.

For example, this issue can be reproduced when riscv32-unknown-elf-objdump is not available on the path. Riscof prints "ERROR | riscv32-unknown-elf-objdump: executable not found. Please check environment setup.", then returns a success code. The responsible code is

            if shutil.which(objdump) is None:
                logger.error(objdump+": executable not found. Please check environment setup.")
                raise SystemExit

It looks like this is caused by raise SystemExit:

$ python
Python 3.11.5 (main, Sep  2 2023, 14:16:33) [GCC 13.2.1 20230801] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> raise SystemExit
$ echo $?
0

Proposed solution

Replace all raise SystemExit with appropriate raise SystemExit(1) or sys.exit(1) calls. Review code in other plugins for similar mistakes. This relates to #64