Open bourgeoa opened 3 years ago
I really like this idea! A few considerations I have (maybe for future versions, because I think just this change would be a great improvement already)
I think we need to be able to single out MAY rules, so a server can choose to run the test for the one they implement but not the rest, to that the applicable MAY rules can be run in CI as well.
May be this could be done with an options parameter in function itIs (arg = 'MUST', options) or multiple MAY depending on the complexity
I think it would be best to require the level for itIs - it will require a bit of work behorehand, but will be more explicit from there on. Also, it might be a good idea to have an ID for every test so that you can explicitly toggle each one.
And just to put it out there: maybe we should make 'MUST' and 'SHOULD' required by default, so you'd have to opt-in to skip them. For 'MAY', the tests are skipped by default and you'd have to opt-in to include them;
It would end up something like this:
export function itIs(level, id) {
switch (level) {
case 'SKIP':
return (name, runner) => { it.skip(`${level} ${name}`, runner); }
break;
case 'MUST':
if (
(process.env.SKIP_MUST) ||
(process.env_SKIP_MUST_' + id])
) {{
return (name, runner) => { it.skip(`${level} ${name}`, runner); }
} else {
return (name, runner) => { it(`${level} ${name}`, runner); }
}
break;
case 'SHOULD':
if (
(process.env.SKIP_SHOULD) ||
(process.env['SKIP_SHOULD_' + id])
) {
return (name, runner) => { it.skip(`${level} ${name}`, runner); }
} else {
return (name, runner) => { it(`${level} ${name}`, runner); }
}
break;
case 'MAY':
if (
(process.env.INCLUDE_MAY) ||
(process.env['SKIP_SHOULD_' + id])
) {
return (name, runner) => { it(`${level} ${name}`, runner); }
} else {
return (name, runner) => { it.skip(`${level} ${name}`, runner); }
}
break;
}
return (name, runner) => { it(`${level} ${name}`, runner); }
}
If the ID option is retained, if defined the ID shall be displayed explicitly in the test name
it(${level} ${Id} ${name}, runner)
I agree to require a level.
For practical reason it may be better to keep the ID optional until the tests and spec are stabilised. Changes imply a review of the CI.
The ID could be the link with the specification.
@michielbdejong @ylebre @edwardsph Each test SHOULD be specified as a MUST, SHOULD, MAY so as to know the specification level required. An html report could be generated with the results (report: work to be done)
I tested a quite easy solution around wrapping jest functions with an
itIs
function :itIs('MAY')
(this allow for further reporting)itIs()
describe("Get RDFa", () => { describe("As JSON-LD", () => { .... itIs('MAY')("Triples", async () => { const triples = await asTriples( jsonText,
${testFolderUrl}example.html
, "application/ld+json" ); expect(triples).toIncludeAllMembers(triplesFromHtml); }); }); describe("As Turtle", () => { .... itIs()("Triples", async () => { const triples = await asTriples( text,${testFolderUrl}example.html
, "text/turtle" ); expect(triples).toIncludeAllMembers(triplesFromHtml); }); }); }); describe("GET Turtle", () => { describe("As JSON-LD", () => { ... itIs('MUST')("Triples", async () => { const triples = await asTriples( jsonText,${testFolderUrl}example.ttl
, "application/ld+json" ); expect(triples).toEqual(triplesFromTurtle); }); });PASS test/surface/conneg.test.ts Alice's pod Get RDFa As JSON-LD ○ skipped MAY Triples As Turtle ✓ MUST Triples (13 ms) GET Turtle As JSON-LD ✓ MUST Triples (7 ms) As Turtle ✓ MUST Triples (2 ms) GET JSON-LD As Turtle ✓ MUST Triples (3 ms)
Test Suites: 1 passed, 1 total Tests: 1 skipped, 4 passed, 5 total Snapshots: 0 total Time: 2.291 s Ran all test suites matching /conneg.test.ts/i.