dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 4 forks source link

Report non-zero exit status if zero exit status is expected #432

Closed pdawyndt closed 10 months ago

pdawyndt commented 10 months ago

The behavior of reporting a non-zero exit status seems no long to be correct when a zero exit status is expected. This is how a non-zero exit status used to be reported (example):

image

Now the wrong exit status is no longer reported:

image

I slightly changed the specification of the test suite, but the only issue that might have caused this, is that files are now specified for the individual test cases, where they used to be specified at the unit-level:

- arguments: ["emoji.txt", "🐎", "👞"]
  stdout: "horseshoe\n"
  files:
    - name: "emoji.txt"
      url: "media/workdir/emoji.txt"

However, I guess this is an unrelated issue, caused by some other changes in TESTed.