Empty traces lead to misleadingly successful test cases

[j-hui]

@sedwards-lab and I came across this issue when refactoring the runtime. We temporarily changed the debug statements to empty statements, meaning the program shouldn't actually print anything. This led to unexpectedly optimistic results reported by the regression test suite, with most test cases "passing".

I think this is caused by the fact that the length of the compiled program's execution trace is capping the length of the interpreter's trace. This cap was added to prevent the interpreter from running forever, but it also means that if the compiled program prints nothing, the interpreter will yield nothing, leading to misleadingly vacuous equivalence.

I'm marking this as wontfix because we will only encounter this if we make any changes that removes the compiled program's output entirely, but I'm noting this anyway because it may lead to some really misleadingly positive test cases, e.g., if we run tests with an incorrect platform that produces no output.

[Rewbert]

I mentioned this briefly to you when I found those two bugs earlier this week, I think.

The issue is that when we query the interpreter for its output we let the C code dictate the length of the trace, while we don't yet know that the C-code does the right thing. When @koengit & I spoke about this he said that what we should do here is run the test twice. Once where we do what we do now, and once where we ask the interpreter for the initial trace and then use this to query the C-code for its trace.

[j-hui]

A very crude, simple fix to this is simply to take a constant number more from the interpreter's event trace than emitted by the compiled code. This won't affect equivalent traces, but will allow the test framework to "reveal" extraneous events for unequal traces.