Upgrade to version 3 spec

[ErikSchierboom]

If possible, this test runner should be updated to version 3 of the test runner interface specification. In version 3, one additional feature is enabled: the ability to link individual tests to tasks. This allows the website to show which tests belong to which tasks.

The way tests are linked to tasks is via an (optional) task_id field, which is an integer that matches the number of the task as defined in the exercise's instructions.md file (note: the instructions start at index 1).

This is an example of a test in the results.json file:

{
  "name": "Expected oven time in minutes",
  "status": "pass",
  "task_id": 1,
  "test_code": "Assert.Equal(40, Lasagna.ExpectedMinutesInOven());"
}

You are completely free in how to implement this. Some options are:

1. Add metadata to a test that the test runner can then discover while running the tests (e.g. an attribute or annotation)
2. Define a test name/task id mapping (e.g. in the exercise's .meta/config.json file)
3. Any other option you can think of...

As this test runner currently implements version 1 of the test runner specification, it might be good to first upgrade to version 2 of the test runner specification before upgrading to version 3.

Let me know if there are any questions.

[James-N-M]

I'll pick this one up ! I guess the first step of this would be a PR to upgrade it to Version 2 then come back to this to get it to version 3.

[James-N-M]

@neenjaw Hey ! I was just wondering if I could get some guidance on this issue.

[neenjaw]

Hey @James-N-M, sorry, I didn't see your first message last week, thanks for pinging again! 💙

I'm pretty sure this test runner mostly implements version 2 specs, as we do per-test reporting. The biggest challenge of fully implementing version 2 is providing the student with the code that is run for each test case. So for example we have a test case like this:

    public function testSingleZeroReturnsNull(): void
    {
        $this->assertEquals(null, rebase(10, [0], 2));
    }

we need to add to the json output the code that has been run, in this case: $this->assertEquals(null, rebase(10, [0], 2));

--

How this is often done in other tracks that have implemented version is this:

• When a student's solution is submitted to the test runner, the test runner is provided with the solution and a copy of the test suite for that exercise
• some static analysis is done to parse the test file to get the code for each test case
• then the outputted json data from the test runner is merged with the results of the static analysis to preduce the version 2 (and 3) spec

So I think probably the first step is to investigate how you might be able to do this programmatically in PHP.

I'm not super familiar with PHP's AST parsing tools, but they must exists and I think phpcs and rector both use them in their code transformations/formatting, but if you can traverse the AST then pluck out the code, format it, that is the crux of the problem.

What do you think? I think there are probably a bunch of ways to do this, but interested to hear your thoughts

[James-N-M]

I'm thinking I'm gonna have to do some research on

• AST parsing tools.
• phpcs
• rector

Also I'm going to look at how the other tracks that have upgraded to v2 and v3 have done this.... perhaps another scripting language. Again thanks for getting back to me :) !

[neenjaw]

Sounds good. The test runner for PHP is coordinated by a bash script, so we can just plug in the addition to analyze the test suite/mutate the output into the pipeline.

[neenjaw]

I wonder if something like https://mattstauffer.com/blog/creating-custom-requires-annotations-for-phpunit/ would work for this. I'm not sure how it might then be ejected to junit, but might be another method to look in to.

[homersimpsons]

Should we assume this is done with https://github.com/exercism/php-test-runner/pull/80 ?

(Note that I wanted to use another approach based on https://docs.phpunit.de/en/10.3/extending-phpunit.html)

[neenjaw]

This is all done