Prefer soft fail over hard fail

[Helijin]

File framework/test.py in compare_signature() does a hard fail if the signature file doesn't exist. For my case, after 30+ hours of simulation, all but one of the tests produced signature files. For the one that didn't, the simulator crashed thereby not generating a signature file. Due to the hard fail, the report is not generated.

Recommending soft fail with missing signature file result indicated in the report so the results of the other 400+ tests can be observed in the HTML report.

[neelgala]

Could you point to the code where you see this hard-fail? as far as I know if the tests fail (or sim fails and no signature file is generated by DUT) the reports are always generated. Could provide a trace-log if possible.

[Helijin]

Sure.

Around line 42 of framework/test.py:

if not os.path.exists(file1) :
    logger.error('Signature file : ' + file1 + ' does not exist')
    raise SystemExit(1)

This code does a hard fail with the SystemExit() if the signature file doesn't get generated for a test case because my simulator crashes. It would be better if it just logged an error in the report that the file wasn't generated so I could still observe the results of the other tests where signature files were generated.

I don't have the trace-log anymore and have moved on to another project, but this above message is the last thing I see in the output and no report is generated that I can see. RISCOF just quits.

Thank you for looking. James

[neelgala]

thanks for this. I will make the necessary patch. Instead of just Pass/Fail I think riscof should have a status "Unavailable" to indicate other reasons for failure

[misprit7]

Just wanted to say that a soft fail would also be much preferable for my application as well, it's quite frustrating when a single test failing to output a signature causes no report to be generated, making it hard to debug what went wrong with my testing framework.