search

dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 5 forks source link

Unexpected exceptions raised in main call may exposed TESTed internals #542

Open pdawyndt opened 3 weeks ago

pdawyndt commented 3 weeks ago

This test suite should check if a call to the function generate_division_by_zero raises a ZeroDivisionError.

- statement: 'generate_division_by_zero()'
  exception:
    message: "division by zero"
    types:
      python: ZeroDivisionError

However, if the following solution is submitted

print(1/0)

The stack trace exposes some of the TESTed internals that should remain hidden for students:

image

pdawyndt commented 2 weeks ago

I guess we just forgot to clean the stack trace of the main call, because what happens above is as expected (test plan first executed main call and then calls a function, but the main call throws an exception that is displayed by the Python runtime on stderr, and exits, probably with a non-zero exit status that is not shown here). So all we should do is hide the execution_0.puy lines in the stack trace.