Support language-specific literals, expressions and statements

[pdawyndt]

TESTed currently only supports language-agnostic expressions and statements to achieve the goal of programming-language agnostic test cases.

However, TESTed-DSL might also be used as a common format for specifying test suites, even for programming exercises that only target a single programming language or a limited list of programming languages. We could also support those use cases.

For exercises that target a single programming language, we could add a top-level setting programming_language: '<language>' to set a specific programming language. Its default value (e.g. 'abstract') is the current behavior: all expected return values (return), expressions (expression) and statements (statement) are interpreted as expressed in the abstract programming language of TESTed-DSL and translated on-the-fly into the programming language of the submission (or oracle). However, if the programming-language attribute is for example set to javascript, the test suite becomes JavaScript-specific, and all return values, expressions and statements are interpreted as expressed in JavaScript. This would obviate the need to translate the values, expression and statements in another language, and allow using language-specific syntax/constructs that are not supported in the abstract programming language of TESTed (e.g. anonymous functions).

For exercises that target a limited list of programming languages, we might provide language-specific representations for all target language at all places where the language-agnostic representation is not supported. Something like

- cases:
  # language-specific
  - expression:
      'python': "sort_words(['SPAM', 'eggs', 'bacon'], lambda word: word.upper())"
      'javascript': 'sortWords(["SPAM", "eggs", "bacon"], word => word.toUpperCase())'
  # language-agnostic
  - return: ['bacon', 'eggs', 'SPAM']

This would allow mixing language-specific and language-agnostic values, expressions and statements in a single test suite. If there's a need, wherever language-specific representations are used, we could also add an 'abstract' representation that is used for all cases where no language-specific representation is given.

[pdawyndt]

Another use case I'm thinking of is the following. If a teacher specifies test suite for a single programming language, the entry-level is lower as the teacher can use a familiar programming language instead of the abstract programming language of TESTed-DSL.

If later it turns out that the exercise could be used across programming language, TESTed could have an option to automatically translate the language-specific test suite into a language-agnostic test suite. TESTed now only needs a way to transform expressions/statement expressed in its abstract programming language into a language-specific format. Its language modules do not support the reverse transformation. However, if that reverse transformation would be supported as well (for some commonly used languages), this use case could be achieved. During the process, the conversion to the abstract language could then check if all language-specific expressions/statements can be converted, and report where non-supported constructs have been used.

[niknetniko]

While thinking about implementing this, one potential pitfall with this approach is that it will require more thought to use when calling code from the submission.

Currently, in most languages, the code of the submission is not imported into the main namespace; instead it resides under a "submission" namespace (for example, in Python we do import submission). With the abstract syntax, this is not an issue, since we know what the function calls are, so we prefix them with the namespace of the submission.

However, when using language-specific literals, these are a "black box" to TESTed: we would need to just insert them into the generated code verbatim. In the example above, you would need to write submission.sort_words(). This is annoying, but not a big issue.

The same issue also occurs for return values: since some languages have "void" return types, we cannot wrap the expression with our function to capture the return value. For example, captureValue(someExpression) will not compile in Java if somExpression is a function call with return type void.

In the abstract syntax, we know if an expression is a function call, so we can work around it. However, in with language-specific literals, we won't know this, so this will potentially result in compilation errors. For this case, there is no easy way of fixing this.

So either we need to provide a way to provide this metadata somehow (e.g. it is a function call from the submission), or document these caveats somewhere. I think for now the second option is probably best, as it is less work, and I don't currently see a nice way to do the first.