exercism / haskell-test-runner

GNU Affero General Public License v3.0
1 stars 13 forks source link

Look for test name to task id mapping in manifest #108

Open fredefox opened 6 months ago

fredefox commented 6 months ago

This is an initial stab at #19. It looks for a name to task id mapping in the projects manifest file metadata.json. For instance for leap the root object should be amended with an object like this:

  "tests": {
    "2015 - year not divisible by 4 in common year": { "task_id": 1 },
    "1970 - year divisible by 2, not divisible by 4 in common year": { "task_id": 1 },
    "1996 - year divisible by 4, not divisible by 100 in leap year": { "task_id": 1 },
    "1960 - year divisible by 4 and 5 is still a leap year": { "task_id": 1 },
    "2100 - year divisible by 100, not divisible by 400 in common year": { "task_id": 1 },
    "1900 - year divisible by 100 but not by 3 is still not a leap year": { "task_id": 1 },
    "2000 - year divisible by 400 in leap year": { "task_id": 1 },
    "2400 - year divisible by 400 but not by 125 is still a leap year": { "task_id": 1 },
    "1800 - year divisible by 200, not divisible by 400 in common year": { "task_id": 1 }
  }

Or whatever the task id is supposed to be. This wasn't entirely clear to me.

I've added a module Manifest.hs that will look for metadata.json. The mechanism used for locating the file simply assumes that the cwd is where .exercism is located. This might not be the most robust way to do it, but it is what the test runner seems to do. So I don't see any immediate benefit to improving this. If metadata.json is not located or fails to parse the task IDs are simply set to null in the generated report.

Here is an example of how the output report changes with this patch for the leap task if you were to add an appropriate section in metadata.json:

 {
     "message": null,
     "status": "fail",
     "tests": [
         {
             "message": null,
             "name": "2015 - year not divisible by 4 in common year",
-            "status": "pass"
+            "status": "pass",
+            "task_id": null
         },
         {
             "message": null,
             "name": "1970 - year divisible by 2, not divisible by 4 in common year",
-            "status": "pass"
+            "status": "pass",
+            "task_id": null
         },
...

I've updated the tests. I had one issue with the tests where the GHC/Stack output seems to have changed wording subtly according to:

-Stack encountered the following errors:
+Stack encountered the error:

I had to add this change to get the tests to work with my local setup, though I appreciate we might need to not include it in the patch or do something else to accommodate the discrepancy.

github-actions[bot] commented 6 months ago

Hello. Thanks for opening a PR on Exercism 🙂

We ask that all changes to Exercism are discussed on our Community Forum before being opened on GitHub. To enforce this, we automatically close all PRs that are submitted. That doesn't mean your PR is rejected but that we want the initial discussion about it to happen on our forum where a wide range of key contributors across the Exercism ecosystem can weigh in.

You can use this link to copy this into a new topic on the forum. If we decide the PR is appropriate, we'll reopen it and continue with it, so please don't delete your local branch.

If you're interested in learning more about this auto-responder, please read this blog post.


Note: If this PR has been pre-approved, please link back to this PR on the forum thread and a maintainer or staff member will reopen it.