dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 5 forks source link

Update default interpretation of YAML strings and objects #457

Closed niknetniko closed 10 months ago

niknetniko commented 10 months ago

YAML strings and objects are now interpreted as literal YAML values by default.

We introduce !expression and !oracle to "cast" strings and objects to expressions and oracles respectively. The list of arguments in an oracle uses the same logic: YAML types by default, expressions when tagged as such.

Using the same logic for arguments is not that obvious: we could also force the use of the expression syntax, as we do with actual expressions and statements. However, it felt more natural to be consistent with the return value (which is in the same YAML object) than with the expressions and statements.

This is basically a reversal of the previous behaviour, where strings and objects needed a !v tag. This tag has now been removed.

Fixes #447. Related: #427.

Not to be merged until the migrations in the repositories are ready (or there is time at least).

After merging and migration, we need to update the docs and the template repository.

niknetniko commented 10 months ago

All exercise repositories have been migrated on Dodona and changes pushed where possible. For posterity, this is the migration script that was run:

migrate.py ```python import re import shlex import subprocess from pathlib import Path print("*** Migrating stuff ***") print(f"Working inside {Path.cwd()}.") cwd = Path.cwd() all_contents = sorted(cwd.iterdir()) excluded_repositories = ["TESTed_challenges.git", "javascript-oefeningen"] start_index = 0 for i, potential_repo in enumerate(all_contents[start_index:], start=start_index): if not potential_repo.is_dir(): print(f"Skipping {potential_repo} as it is not a directory.") continue if potential_repo.name in excluded_repositories: print(f"Skipping {potential_repo} as it is excluded from this script.") continue print(f"Entering repository {potential_repo}...") potential_testsuites_one = list(potential_repo.rglob("evaluation/*.yaml")) potential_testsuites_two = list(potential_repo.rglob("evaluation/*.yml")) potential_testsuites_one.extend(potential_testsuites_two) potential_testsuites = sorted(potential_testsuites_one) print(f" Found {len(potential_testsuites)} suites") changed = [] for potential_suite in potential_testsuites: print(f" [{i}] Migrating {potential_suite}...") with open(potential_suite, "r") as sources: lines = sources.readlines() new_lines = [] for line in lines: nl = re.sub("return: *([\"'])+", "return: !expression \1", line) nl = re.sub("return: *$", "return: !oracle", nl) nl = re.sub("return: {", "return: !oracle", nl) nl = nl.replace("return: !value", "return:") nl = nl.replace("return: !v", "return:") if nl != line: print(f" Converted {line!r}") print(f" To: {nl!r}") new_lines.append(nl) if new_lines != lines: inp = input("Press enter to continue... ") if inp != "": print( f"Aborted. Restore repo before running again and set start index to {i-1}" ) exit() changed.append(potential_suite) with open(potential_suite, "w") as destination: destination.write("\n".join(new_lines)) if changed: print(f" {len(changed)} files were changed...") choice = input( "Ready to commit and push? Leave empty to confirm, type something otherwise" ) if choice == "": subprocess.run(["git", "pull"], check=True, cwd=potential_repo) s = " ".join( [shlex.quote(str(x.relative_to(potential_repo))) for x in changed] ) command = f"git add {s}" print(command) subprocess.run(command, shell=True, check=True, cwd=potential_repo) subprocess.run( [ "git", "commit", "-m", "Migrate to new syntax for returns", "-m", "See https://github.com/dodona-edu/universal-judge/pull/457 for more information", "--no-verify", ], cwd=potential_repo, check=True, ) subprocess.run(["git", "push"], check=True, cwd=potential_repo) else: print( "Aborting commit. Next time, update the starting index to prevent duplicate processing" ) exit() ```