Return a non-zero exit code, when testsuite has failing tests

[Paebbels]

Currently, a testsuite with failing tests returns an exit code of 0.

Proposed exit codes:

• 0 - all OK: analysis, simulation, report generation (TCL after scripts)
• 1 - analysis failed with compilation errors from vcom, vlog, ...
  (see #15)
• 2 - at least one simulation failed with errors
• 3 - all simulations are OK, but there have been errors in after scripts (e.g. report generations)

Maybe an option is required to configure if an overall run should fail if after-scripts had errors.
SetIgnoreErrorsInReporting true

[JimLewis]

How about test case failures? IE test ran but had failures?

TCL suggests the following return values: exit(0) on passed exit(1) on normal error - ie test case errors - exit(2) on script ended abnormally exit(3) failed during analyze exit(4) failed during simulate

What does CI do with non-zero return values? In particular if it was a test case error, I still want it to generate its report. Maybe even if it has a simulate or analyze error - did anything complete and if so, produce what reports you can would be most valuable.

Scripts failing during report generation is a bug and is no more or less likely than other bugs in the scripts prior to that point. We can catch them, but I doubt we can accurately bin them.

[Paebbels]

I think you address this issue, right?

[JimLewis]

Done. Documentation coming in 2022.09.