search

rubensworks / rdf-test-suite.js

Executes the RDF and SPARQL test suites.
MIT License
9 stars 6 forks source link

Error if running zero tests #102

Open RubenVerborgh opened 1 month ago

RubenVerborgh commented 1 month ago

In SPARQL.js, running npm run spec-base-query now silently fails with ”0/0 tests”:

> sparqljs@3.7.1 spec-base-query
> rdf-test-suite spec/parser.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest.ttl -s http://www.w3.org/TR/sparql11-query/ -c .rdf-test-suite-cache/

Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv01.tsv
Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv02.tsv
Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv03.tsv
Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv01.tsv
Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv02.tsv
Error: Could not parse the query result with content type text/tab-separated-values at http://w3c.github.io/rdf-tests/sparql/sparql11/csv-tsv-res/csvtsv03.tsv
✔ 0 / 0 tests succeeded!
rubensworks commented 1 month ago

In comunica we use http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl, could you try that one?

I'm guessing some breaking changes were introduced in http://w3c.github.io/rdf-tests/sparql/sparql11/manifest.ttl for the RDF 1.2 tests.

RubenVerborgh commented 1 month ago

Thanks! That's indeed quite confusing, especially given that they broke URIs repeatedly (https://github.com/RubenVerborgh/SPARQL.js/commit/0ef0d4c3b9c44f1c9e075f8ac6837b34bf728466).

With npx rdf-test-suite spec/parser.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl -s http://www.w3.org/TR/sparql11-query/ -c .rdf-test-suite-cache/, I get: ✔ 301 / 301 tests succeeded! (skipped 198). The 198 still seems like a lot, but perhaps that's okay?

For npx rdf-test-suite spec/parser.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-sparql11-update.ttl -s http://www.w3.org/TR/sparql11-update/ -c .rdf-test-suite-cache/, I unfortunately still have ✔ 0 / 0 tests succeeded!.

Suggestion: still error on 0/0?

rubensworks commented 1 month ago

I get: ✔ 301 / 301 tests succeeded! (skipped 198). The 198 still seems like a lot, but perhaps that's okay?

That should be fine indeed, since the passed runner is configured to skip all non-parsing tests: https://github.com/RubenVerborgh/SPARQL.js/blob/main/spec/parser.js#L4-L15

For npx rdf-test-suite spec/parser.js http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-sparql11-update.ttl -s http://www.w3.org/TR/sparql11-update/ -c .rdf-test-suite-cache/, I unfortunately still have ✔ 0 / 0 tests succeeded!.

http://w3c.github.io/rdf-tests/sparql/sparql11/manifest-all.ttl should also work here. At least it does in comunica: https://github.com/comunica/comunica/blob/master/engines/query-sparql/package.json#L54-L56

RubenVerborgh commented 1 month ago

Thanks, LGTM. Maybe just consider if you want the suite to error if it finds 0 tests.

rubensworks commented 1 month ago

Maybe just consider if you want the suite to error if it finds 0 tests.

Good idea, we should add that.

In addition, we may also want to add a flag that does not error in such cases, as there may be cases where zero tests are valid.