dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 4 forks source link

Pass objects to custom oracles #473

Closed niknetniko closed 7 months ago

niknetniko commented 7 months ago

This replaces the expected/actual values and provides some more context for the oracles.

Fixes #472 and closes #470.

A check function must now look like this:

# noinspection PyUnresolvedReferences
from evaluation_utils import EvaluationResult

def evaluate_no_arguments(context):
    pass

def evaluate_no_arguments(context, arg1, arg2):
    pass

When passing additional arguments in the test suite, they are added to the function call as arg1, arg2, etc.

The context is a class with the following fields:

class ConvertedOracleContext:
    expected: Any  # Expected value from the test suite
    actual: Any        # Value generated by the submission
    execution_directory: str  # Directory where the execution took place (this will be a subdirectory of the workdir)
    evaluation_directory: str  # Path to the evaluation directory (in the exercise directory)
    programming_language: str  # Name of the programming language of the submission
    natural_language: str  # Natural language