Aalto-LeTech / JAAL

A data format for recording answers to JSAV exercises
0 stars 1 forks source link

test.js: Invalid schema error message validation #17

Closed Meratyn closed 2 years ago

Meratyn commented 2 years ago

Fulfill #15 Expand invalid tests with a verification of expected failure. Invalid tests have two new fields: errorInstancePath and errorMessage. These contain the expected error instance path and message. test.js checks the validation errors against these fields for the invalid test cases.

Error messages for failed test shortened with redundant information removed.

atilante commented 2 years ago

I tested the testing framework:

  1. Ensure that the testing framework produces a message when the validation error message is different than in the *errorMessage" field of the test case.

Modify spec/test/invalid/edge-1node.json:

{
  "description": "An Edge cannot have only one node.",
  "id": "edge1",
  "node": [ "node1", "node2", "node3" ], 
  "errorInstancePath": "/node",
  "errorMessage": "must NOT have fewer than 2 items"
}

Output of test.js referring to the test case:

Test /invalid/edge-1node.json:
     /node must NOT have more than 2 items
Expected error:  /node must NOT have fewer than 2 items

Yes, the test framework catches this case. However, for better usability, the error message could actually be improved. Example:

Test invalid/edge-1node.json:
    Location: /node
    Expected: must NOT have more than 2 items
    Found   : must NOT have fewer than 2 items
  1. Ensure that the testing framework produces a message when the validation error location is different than in the *errorInstancePath" field of the test case. Modify spec/test/invalid/edge-1node.json:
    {
    "description": "An Edge cannot have only one node.",
    "id": "totallyInvalidId",
    "node": [ "node1" ], 
    "errorInstancePath": "/node",
    "errorMessage": "must NOT have fewer than 2 items"
    }

Output of test.js referring to the test case:

Test /invalid/edge-1node.json:
     /id must match pattern "^edge"
Expected error:  /node must NOT have fewer than 2 items

Yes, the test framework catches this case.

I will do the rest of the proposals myself.