Bluesim binaries exit with status 0 on assert failure

[cbiffle]

Bluesim self-checking testbenches using dynamicAssert wind up exiting with status 0 whether they pass or fail. This makes them difficult to integrate into conventional test frameworks (or, for that matter, Unix shell scripts) that use exit status 0 to indicate success, and non-zero to indicate failure.

The main workaround I've seen in the field is arranging for a test to print some special string, such as PASS, and grepping the output for it. This is not great. It would be preferable to return a non-success exit status on assert failure.

[quark17]

You mention specifically Bluesim -- does Verilog sim not have the same problem?

I see that the Assert package defines the fail behavior like this:

$display (assertMessage "Dynamic" s)
$finish 0

The argument to $finish is just telling the simulator what info to print, not the error code. I would guess that the simulator would exit with a success code (whether Verilog or Bluesim). I would guess that we'd want to change this to use $fatal, which does tell the simulator to exit with an error code (of its choosing). [This is an old package, and I don't know how often people use it, so I wouldn't be surprised if it needs some updating or rethinking.]

However, I do see other issues that would prevent a Bluesim sim from exiting with an error code, even if that change were made. For one, Bluesim objects are executed from a Bluetcl script (src/bluetcl/bluesim.tcl) whose last line is exit 0. The script would need to get an exit code from the sim run call and return that. I don't offhand know if the sim run command itself is properly returning the exit code, so that might need adjusting too. (The Bluetcl issues can be bypassed by compiling the model into an executable with a C++ driver rather than driving from Bluetcl. See bsc/util/bsim_standalon/.) For two, the implementation of $fatal in Bluesim (see dollar_fatal() in dollar_display.cxx) seems to be using its numeric argument as an exit code, when I believe it should have the same meaning as for $finish.

[thoughtpolice]

I already work around this using the described method: my simulation bench just emits some unique string like ---- SIMULATION EXIT: 0 SUCCESS and a harness runs them all and ensures that exists. Personally, I find this to be adequate, but in general some method to exit a simulation with a different error code would be useful.

Haskell has the System.Exit API, so perhaps just copying that would be good. But I don't know what the behavior of these APIs should be when you emit Verilog for own your chosen simulator, though. I'm not sure if there are any existing rules about Bluesim-only APIs like that...

[nanavati]

The simplest thing that occurs to me is that if you end your simulation with $fatal or call $error along the way, Bluesim should exit with a non-zero error code. I don't know if any Verilog simulators do that, but if they don't, they should.

[quark17]

When BSC was a proprietary product and BSV the only supported syntax, there was a goal of being similar to Verilog/SystemVerilog when possible -- in this case, to use $fatal and $error instead of inventing a new API. Now that BH/Classic is more supported (and other front-end are possible, if folks want to add them!), we can reconsider that. However, we still have a goal that the different backends should implement the same behavior -- so we should be careful about defining a behavior for Bluesim that can't be equally implemented in generated Verilog. That argues for using $fatal and $error as the API, since they can pass through to the Verilog. Although they are SystemVerilog tasks, not in earlier Verilog standards (I don't believe?), so there might be issues if we want to support generating a purer Verilog output (but we can probably retire the -v95 flag and not worry about it). That might explain why the Assert package uses $finish -- because that may have been the only available task at the time?

As I said, Bluesim needs to be fixed to properly implement $fatal. (And then probably the Assert package ought to change to use it.)