exercism / nim-test-runner

GNU Affero General Public License v3.0
2 stars 3 forks source link

Upgrade to version 3 spec #91

Open ErikSchierboom opened 3 years ago

ErikSchierboom commented 3 years ago

If possible, this test runner should be updated to version 3 of the test runner interface specification. In version 3, one additional feature is enabled: the ability to link individual tests to tasks. This allows the website to show which tests belong to which tasks.

The way tests are linked to tasks is via an (optional) task_id field, which is an integer that matches the number of the task as defined in the exercise's instructions.md file (note: the instructions start at index 1).

This is an example of a test in the results.json file:

{
  "name": "Expected oven time in minutes",
  "status": "pass",
  "task_id": 1,
  "test_code": "Assert.Equal(40, Lasagna.ExpectedMinutesInOven());"
}

You are completely free in how to implement this. Some options are:

  1. Add metadata to a test that the test runner can then discover while running the tests (e.g. an attribute or annotation)
  2. Define a test name/task id mapping (e.g. in the exercise's .meta/config.json file)
  3. Any other option you can think of...

Let me know if there are any questions.

ynfle commented 2 years ago

Any suggestions for tagging exercises?

3 possibilities I see:

  1. doc comments and process them ourselves
  2. Test name/suite names
  3. Some custom pragma of sorts

The downsides:

  1. 🤷‍♂️
  2. Changing test names would pollute error messages but suite names sounds like a good idea
  3. Confusing to end users, as that wouldn't be portable to new projects and a new concept
ynfle commented 2 years ago

@ee7 thoughts?