dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 5 forks source link

Clarify `context not executed` message #342

Closed pdawyndt closed 1 year ago

pdawyndt commented 1 year ago

Submission: 13852943

Not clear for student what this message means, and what causes the message.

If I run the first test case locally, this is the stack trace of the exception that is raised

54:5 Uncaught RangeError: Invalid string length
    at hoofdletter (<code>:5:25)
    at groepen (<code>:13:24)

so student should get to see this message as feedback.

pdawyndt commented 1 year ago

@niknetniko Thanks for solving the underlying issue of not properly reporting the "memory limit exceeded" case.

I still believe Dodona should have a better way to report test cases and contexts that have not been executed and evaluated by the judge due to some critical event (e.g. an exceeded time or memory limit in earlier contexts/test cases).

image

Using the more granular status (correct, wrong, runtime error, time limit exceeded, memory limit exceeded) instead of the binary status (correct/wrong) in rendering the feedback could improve the interpretability for the students by better explaining why something went wrong. We could use different color codes and different icons to make this differentiation, and extend the more granular status with an additional "not executed/evaluated" status.

image

Bonus: If no output (in the above case return value) is generated, we could avoid that Dodona puts effort in diffing the expected/generated output.