exercism / x86-64-assembly-test-runner

GNU Affero General Public License v3.0
4 stars 4 forks source link

Upgrade to version 2 spec #19

Open ErikSchierboom opened 3 years ago

ErikSchierboom commented 3 years ago

We've recently updated the test runner interface specification to allow for two types of test runners, identified via a new version property in the results.json file (see the spec for the version property):

  1. version: 1 test runners are quite basic, and only detect if either all tests passed or not. These test runners capture the console output of the test runner and output that. They do not include details on individual tests passing/failing
  2. version: 2 test runners contain detailed information on individual tests passing/failing. We've recently added a required field named test_code, which contains the code the test ran to verify the behavior (see the spec for the test_code property).

The test runner's output is currently what I refer to as version 1.5: it contains the individual test information, but does not yet contain the test_code key. If possible, the test runner should be updated to also include the test code, in which case the version property should also be added with 2 as its value.

You are completely free on how you want to extract the test code, either via the AST or via finding the test code in the source code text via string manipulation.

Let me know if there are any questions.

hut8 commented 2 years ago

The version key is present now, so is that why tests.sh fails? I'm trying to test the runner locally and it fails for seemingly this reason.

ErikSchierboom commented 2 years ago

@hut8 Yep. I've just opened https://github.com/exercism/x86-64-assembly-test-runner/pull/34 to fix this.

hut8 commented 2 years ago

@ErikSchierboom you're awesome. Thanks for your work and quick reply. πŸ‘πŸ»

ErikSchierboom commented 2 years ago

@hut8 In case you hadn't seen, the PR was merged so this should be fixed now.

hut8 commented 2 years ago

πŸ‘πŸ» Saw it. This can be closed then, right?

ErikSchierboom commented 2 years ago

Not entirely, as this bit has not been implemented:

If possible, the test runner should be updated to also include the test code