dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 5 forks source link

Invalid DSL when using assignment with function call #544

Open BrentBlanckaert opened 1 month ago

BrentBlanckaert commented 1 month ago

When trying to make an exercise with a statement that has an assignment using a function call (as in the example below), You will get an invalid DSL error with no extra info. I was able to track a part of the problem down to here. In this case the type would FunctionType.FUNCTION.

- tab: sortWords
  contexts:
    - testcases:
      - statement: 'result = countWords(["Gobber", "Hiccup", "Astrid", "Stoick", "Toothless"], "httyd.txt")'
      - expression: 'sortWords(result)'
        return: ["Hiccup", "Toothless", "Stoick", "Astrid", "Gobber"]
pdawyndt commented 1 month ago

Since TESTed has to infer the type of the variable, we may need to add type information about the return value of the function. No idea how to specify that information in the DSL, so I'll ask @niknetniko.

Could you try doing it this way, so with an explicit int type conversion (guessing that the function should return a dict):

- tab: sortWords
  contexts:
    - testcases:
      - statement: 'result = dict(countWords(["Gobber", "Hiccup", "Astrid", "Stoick", "Toothless"], "httyd.txt"))'
      - expression: 'sortWords(result)'
        return: ["Hiccup", "Toothless", "Stoick", "Astrid", "Gobber"]
niknetniko commented 1 month ago

One way is to use Python type annotations:

- tab: sortWords
  contexts:
    - testcases:
    - statement: 'result: map = count_words(["Gobber", "Hiccup", "Astrid", "Stoick", "Toothless"], "httyd.txt")'
                      # ^^^^^ add this
    - expression: 'sort_words(result)'
  return: ["Hiccup", "Toothless", "Stoick", "Astrid", "Gobber"]

Also note I have used snake case for the function names; otherwise TESTed should print warnings.

When I run the original test suite locally, I do get an error which says what is wrong, so not sure why did not get that:

Traceback (most recent call last):
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/ast_translator.py", line 354, in parse_string
    return _translate_to_ast(tree, is_return)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/ast_translator.py", line 333, in _translate_to_ast
    return _convert_statement(statement_or_expression)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/ast_translator.py", line 312, in _convert_statement
    return _convert_assignment(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/ast_translator.py", line 147, in _convert_assignment
    raise InvalidDslError(
tested.dsl.ast_translator.InvalidDslError: Could not deduce the type of variable result: add a type annotation.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/niko/Ontwikkeling/universal-judge/tested/manual.py", line 57, in <module>
    run(config, sys.stdout)
  File "/home/niko/Ontwikkeling/universal-judge/tested/main.py", line 33, in run
    suite = parse_dsl(textual_suite)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/translate_parser.py", line 761, in parse_dsl
    return _convert_dsl(dsl_object)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/translate_parser.py", line 742, in _convert_dsl
    tabs = _convert_dsl_list(tab_list, context, _convert_tab)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/translate_parser.py", line 715, in _convert_dsl_list
    objects.append(converter(dsl_object, context))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/translate_parser.py", line 682, in _convert_tab
    contexts = _convert_dsl_list(tab["contexts"], context, _convert_context)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/translate_parser.py", line 715, in _convert_dsl_list
    objects.append(converter(dsl_object, context))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/translate_parser.py", line 663, in _convert_context
    testcases = _convert_dsl_list(raw_testcases, dsl_context, _convert_testcase)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/translate_parser.py", line 715, in _convert_dsl_list
    objects.append(converter(dsl_object, context))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/translate_parser.py", line 586, in _convert_testcase
    the_input = parse_string(expr_stmt)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/niko/Ontwikkeling/universal-judge/tested/dsl/ast_translator.py", line 356, in parse_string
    raise InvalidDslError("Invalid DSL") from e
tested.dsl.ast_translator.InvalidDslError: Invalid DSL