Build Test Runner

[ErikSchierboom]

This issue is part of the migration to v3. You can read full details about the various changes here.

In Exercism v3, one of the biggest changes is that we'll automatically check if a submitted solution passes all the tests.

We'll check this via a new, track-specific tool: the Test Runner. Each test runner is track-specific. When a new solution is submitted, we run the track's test runner, which outputs a JSON file that describes the test results.

The test runner must be able to run the tests suites of both Concept Exercises and Practice Exercises. Depending on the test runner implementation, this could mean having to update the Practice Exercises to the format expected by the test runner.

Goal

Build a test runner for your track according to the spec.

If you are building a test runner from scratch, we have a starting guide and a generic test runner that can be used as the base for the new test runner.

If a test runner has already been built for this track, please check if it works on both Concept Exercises and Practice Exercises.

It can be very useful to check how other tracks have implemented their test runner.

Tracking

https://github.com/exercism/v3-launch/issues/4

[ghost]

Must the test-runnner be a stand-alone thing. For Pharo this can be difficult because Pharo is image -based

@ErikSchierboom

[ErikSchierboom]

@RoelofWobben Yes it must be a separate thing. It is okay if a track does not have a test runner, but the experience for users on the website will be less useful (the online editor will be disabled). For Pharo that might not matter much due to already having its own editing environment.

[ghost]

oke

What I think easily can be done is to include a sort of test-runner in the image. So the json file is included in the submission. I know that Tim has done something like that because When you submit you get on v2 also a text-file with the test-results is included.

That the online editor is not working is not a problem at all. But I do not like that when all tests are green , the user still sees failed on the website. Could be very confusing.

[Bajger]

Hi @ErikSchierboom ! I tried to look at the spec link (https://github.com/exercism/v3-docs/blob/main/anatomy/track-tooling/test-runners/interface.md) and starting guide (https://github.com/exercism/v3-docs/blob/main/anatomy/track-tooling/test-runners/creating-from-scratch.md) but links are invalid. Could you fix them? Thanks! (Honestly, I may create a bug report, but I don't know in which repo exactly). Also, there is a way to run tests through github actions for Pharo image using SmalltalkCI project, which might be what we want. https://github.com/hpi-swa/smalltalkCI#testcase-selection Question is: how exactly we could load user-specific code from Exercism submissions (determined by CLI-token) to prepared Pharo image prior running the test cases.

[ErikSchierboom]

@Bajger The correct repo for the docs is https://github.com/exercisn/docs, which contents are rendered on https://exercism.org/docs. I'd link to the documents on the https://exercism.org/docs/ pages, so:

The interface: https://exercism.org/docs/building/tooling/test-runners/interface The specification: https://exercism.org/docs/building/tooling/test-runners/docker

Could you submit a PR? I'm awfully busy right now.

Also, there is a way to run tests through github actions for Pharo image using SmalltalkCI project, which might be what we want. https://github.com/hpi-swa/smalltalkCI#testcase-selection

This might be good for inspiration, but the test runner can't call a github action as the test runner does not have an internet connection when running.

Question is: how exactly we could load user-specific code from Exercism submissions (determined by CLI-token) to prepared Pharo image prior running the test cases.

I have no idea how that would work because I know virtually nothing about Pharo, sorry.