dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 5 forks source link

Feat/add comments #555

Open BrentBlanckaert opened 1 month ago

BrentBlanckaert commented 1 month ago

Issue #371

Adding line comments for statements and expressions.

Example:

Test suite:

- tab: sort_words
  contexts:
    - testcases:
      - statement: 'result: map = count_words(["Adam", "Barry", "Vanessa", "Ken"], "bee.txt") # This is a statement'
      - expression: 'sort_words(result) # This is an expression'
        return: ["Barry", "Vanessa", "Ken", "Adam"]
        files:
          - name: "bee.txt"
            url: "bee.txt"

Output:

{"command": "start-judgement"}
{"title": "sort_words", "command": "start-tab"}
{"command": "start-context"}
{"description": {"description": "result = count_words(['Adam', 'Barry', 'Vanessa', 'Ken'], 'bee.txt') # This is a statement", "format": "python"}, "command": "start-testcase"}
{"command": "close-testcase"}
{"description": {"description": "sort_words(result) # This is an expression", "format": "python"}, "command": "start-testcase"}
{"expected": "['Barry', 'Vanessa', 'Ken', 'Adam']", "channel": "return", "command": "start-test"}
{"generated": "['Barry', 'Vanessa', 'Ken', 'Adam']", "status": {"enum": "correct"}, "command": "close-test"}
{"command": "close-testcase"}
{"message": {"description": "<div class='contains-file''><p>File: <a href=\"bee.txt\" class=\"file-link\" target=\"_blank\"><span class=\"code\">bee.txt</span></a></p></div>", "format": "html"}, "command": "append-message"}
{"data": {"statements": "result = count_words(['Adam', 'Barry', 'Vanessa', 'Ken'], 'bee.txt')\nsort_words(result)"}, "command": "close-context"}
{"command": "close-tab"}
{"command": "close-judgement"}
BrentBlanckaert commented 1 week ago

I've noticed something else that might have to be accounted for. In th global test a description is used. Like the following:

- tab: "Global variable"
  testcases:
    - expression: "GLOBAL_VAR # The name of the global variable"
      return: "GLOBAL"
      description:
        description: "Hallo"
        format: "code"

Instead of getting:

{"description": {"description": "GLOBAL_VAR # The name of the global variable", "format": "code"}, "command": "start-testcase"}

I get the following instead:

{"description": {"description": "Hallo # The name of the global variable", "format": "code"}, "command": "start-testcase"}

Is this the desired outcome or do we want to turn line-comments off in those cases?

pdawyndt commented 1 week ago

I've noticed something else that might have to be accounted for. In th global test a description is used. Like the following:

I think we only want the comment if it is in the description itself. We do not want to migrate comments from the expression or statement that is actually executed to the description as was done in your example.

BrentBlanckaert commented 1 week ago

I think we only want the comment if it is in the description itself. We do not want to migrate comments from the expression or statement that is actually executed to the description as was done in your example.

I'm not sure I'm following. So when comments are in a statement/expression and there is no description, we keep those comment? If there is a desciption we take the comment from that?

BrentBlanckaert commented 1 week ago

Added the ability to add comment to a description too. When a comment is present in the expression, it will be ignored and the comment in the description will be used. However I'm not sure if the description is always valid to be parsed by the tokenizer. I this is not the case then I suggest something like the following:

description:
  description: "Hallo"
  format: "code"
  comment: "This is a greeting"

Right now it simply like this:

description:
  description: "Hallo # This is a greeting"
  format: "code"
BrentBlanckaert commented 1 week ago

Comment can now only put at the end of statements/expressions. Statements and Expressions in the in the TESTed-DSL should only be 1 line so comments in between lines like the on below are irrelevant.

- tab: sort_words
 contexts:
  - testcases:
     - statement: >
              result: map = count_words(["Gobber", # This is a name
                                         "Hiccup", # This is another name
                                         "Astrid", # This yet another name
                                         "Stoick", # WOOOO!!! A name!
                                         "Toothless" # and the last name
                                         ], "httyd.txt") # This is a statement
     - expression: 'sort_words(result) # This is an expression'
       return: ["Hiccup", "Toothless", "Stoick", "Astrid", "Gobber"]
       files:
         - name: "httyd.txt"
           url: "httyd.txt"

Furthermore, block-comment don't need to be supported by TESTed either. Some Languages might not even support it.

When a description is present, the comments from the corresponding expression should not be pasted over. You should write the comments yourself. The description is never parsed so you can use whatever symbol you want to start a comment.