If the test file and the user's solution both compile successfully, a runtime problem like an uncaught exception should produce:
a top-level status of fail.
a test-level status of error.
However, in such cases our test-level status was fail. This PR changes it to error, and adds a test.
I think we can make the specification clearer on this point.
@iHiD quote from Slack:
If the solution fails to compile or the tests can’t run for some reason, then the top level status should be error. If the tests can run, then the top level status should be pass or fail. So if a test runs but errors, that should result in the top level status being fail.
To put it another way, in the UI, if we get a top level error, we’ll only show the top level message, not any of the test fails/errors.
If the test file and the user's solution both compile successfully, a runtime problem like an uncaught exception should produce:
statusoffail.statusoferror.However, in such cases our test-level
statuswasfail. This PR changes it toerror, and adds a test.I think we can make the specification clearer on this point.
@iHiD quote from Slack: