dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 4 forks source link

Make arguments of oracle consistent with return #403

Closed pdawyndt closed 10 months ago

pdawyndt commented 11 months ago

An expected return value is a strongly typed object that can be expressed in YAML representation (return) or in string representation (return_raw).

The extra arguments of oracle (in addition to the generated and expected value) are also strongly typed objects. To be consistent with return values, this should be expressed as a sequence of YAML respresentations (arguments) or as a sequence of string representations (arguments_raw). Now we only have arguments, which actually takes what should be passed to arguments_raw’.

pdawyndt commented 11 months ago

I'm not really pleased with the _raw suffix to make a distinction between YAML representation (return, arguments) using YAML objects and string representation (return_raw, 'arguments_raw') using the abstract or specific programming language. Can't think of a good alternative for now that indicates "code fragment expressed in abstract/specific programming language".

If we would manage to extend the YAML parser so that (only) TESTed data types can be used as !!-types, we could also create a special (non-TESTed) type raw (sticking to that name for now) that would allow to write return: !!raw 'string' as an alternative for return_raw: 'string'. Maybe we don't even want the return_raw attribute anymore if this would be possible.

pdawyndt commented 11 months ago

Even better: the only time where a unified return may not explicitly differentiate between YAML representation and string representation is when its value is a string. The default interpretation of such a string could be as the string representation. As such, return, expression and statement all have string representations as values. To interpret as a string as a literal string in YAML representation, we could use an explicit !!string cast.

Same strategy could also be applied to arguments, where each string argument would need its own !!string cast if all are given in YAML representation.