search

exercism / java-test-runner

GNU Affero General Public License v3.0
9 stars 13 forks source link

Never use status "error" when tests have been run #64

Closed sanderploegsma closed 1 year ago

sanderploegsma commented 1 year ago

After reading this forum topic I realized that the Java test runner incorrectly uses the status error when any test case has errored. This is not in line with the spec, which basically says that it should only be used if there was a global error running the tests.

So with this update, any test case having a status error will result in a overall status fail.

ErikSchierboom commented 1 year ago

Should there maybe be a test case to catch this: https://github.com/exercism/java-test-runner/tree/main/tests? Or is it already caught

sanderploegsma commented 1 year ago

I was thinking about this, but currently I don't think it's even a state we can reach. JUnit gives us the statuses SUCCESSFUL, FAILED and ABORTED, which map to pass, fail and error respectively. So to test for this we would have to cancel a test case in-flight, which is not something we can easily do in CI.