dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 5 forks source link

Add support for custom checks in DSL #393

Closed niknetniko closed 1 year ago

niknetniko commented 1 year ago

This adds support for custom checks in the DSL:

- tab: 'Test'
  contexts:
    - testcases:
        - statement: 'test()'
          return_raw:
            value: "'hallo'"
            evaluator: "custom"
            language: "python"
            file: "test.py"
            name: "evaluate_test"
            arguments: ["'yes'", "5", "set([5, 5])"]

As a reminder, custom checks are available for textual output (e.g. stdout, stderr) and return values. For return values, only return_raw is supported (not return, as we cannot know if an object is a value or a custom check).

Fixes #391.