dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 4 forks source link

Resume if not all test cases were executed due to early exit #431

Open pdawyndt opened 10 months ago

pdawyndt commented 10 months ago

When multiple test cases are executed in a single run, and not all test cases are executed, we might make a distinction whether this is due to a time limit being exceeded or any other reason (e.g. submission exits unexpectedly). In the latter case, it would be better if we execute the remaining test cases in a new run.

Here's an example where students might otherwise see "Context not executed" for reasons they cannot explain.