dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 4 forks source link

Check unexpected return value #372

Closed pdawyndt closed 11 months ago

pdawyndt commented 1 year ago

By mistake, I used return-raw (with a dash) instead of return_raw:

- statement: "sum(1, 2, 3)"
  return-raw: "6"

As a result, TESTed assumed that no return value was expected for this statement (this is OK, as my mistake makes TESTed think that no expected return value is set). However, TESTed also did not evaluate the actual return value, but instead completely ignored it. I would have expected that in this case, TESTed would have shown me that a value was returned, where no value was expected.

pdawyndt commented 1 year ago

Here's another example submission. In this case, the methode horizontaalInvullen is not expected to return anything, but the method returns a reference to the object on which it is called. This is accepted by TESTed, where it shouldn't.

niknetniko commented 1 year ago

While experimenting with #374, I encountered some issues with the concept of disallowing a return value.

The main issue is how to handle "no return value" in a language-agnostic way. For example, in Java, the compiler will enforce this if the function has a return value of "void", but not all languages do this. On the other hand, in languages such as Python and JavaScript, we cannot differentiate between "no return value" and "returning None/undefined".