Upgrade to test runner v3

[sanderploegsma]

What started off as an attempt to speed up the test runner by dropping Maven in favor of directly compiling solutions using Java's built-in JavaCompiler resulted in almost a complete rewrite:

Programmatic compilation of solutions

The test runner now compiles the submitted solutions directly. This loses the runtime overhead of running a Maven process in the background. This is a big time-saver with little extra complexity IMO.

Programmatic invocation of JUnit

With the solutions being compiled by the test runner, we should also programmatically run JUnit. The test runner contains both junit-jupiter-engine and junit-vintage-engine to support both JUnit 4 and JUnit 5 tests, which is useful if we want to gradually adopt JUnit 5 in the exercises.

Capturing of stdout

I configured JUnit to capture stdout while running tests, and made sure the captured output ends up in the JSON output of the test runner. This allows students to add print statements to their solutions in the online editor to debug failing tests if they choose.

V3 compatibility

I came up with the following solution to allow for the task_id to be determined in order to support the v3 spec of the test runner: we can use JUnit's @Tag annotation to annotate each concept exercise test with the appropriate task ID, and then retrieve the annotations from the JUnit test execution results.

The concept exercise tests should then contain tests like this:

public class ConceptExerciseTest {
    @Test
    @Tag("task:1")
    public void testForTask1() {
        // ...
    }

    @Test
    @Tag("task:2")
    public void testForTask2() {
        // ...
    }
}

And it would result in the following output:

{
    "version": 3,
    "status": "pass",
    "tests": [
        {
            "name": "testForTask1()",
            "status": "pass",
            "task_id": 1
        },
        {
            "name": "testForTask2()",
            "status": "pass",
            "task_id": 2
        }
    ]
}

Unfortunately there is nothing of the sort in JUnit 4. It supports categories but since these are class markers and not string markers they would have to be defined separately in every test class, and it would have to rely on some sort of naming convention for the test runner to be able to correctly determine which marker belongs to which task ID. It's possible, but I figured using JUnit 5's @Tag is a lot easier.

[sanderploegsma]

@ErikSchierboom you’ll like this I think 😃

Not sure whether a similar approach will help the Kotlin test runner though as you already mentioned that using kotlinc didn’t show a major difference.

[sanderploegsma]

@ErikSchierboom thanks!

Just wondering, does this PR fall in the size:large or size:huge category in your opinion? 😄

[ErikSchierboom]

Well, it depends how long you were working on it :D