N3 tests - Githubissues

doerthe commented 4 years ago

I would like to know whether the parser accepts the current syntax tests of the N3 working group. If not, which cases fail?

rubensworks commented 4 years ago

@doerthe Do you know if the N3 tests will be compatible with the RDF test manifest vocab? If so, this would make the tests directly usable with many of the existing tools, such as rdf-test-suite, which would allow N3.js to easily run the N3 tests.

RubenVerborgh commented 4 years ago

Actually, yes, I'm going to wait for that.

doerthe commented 4 years ago

So far, the tests are not compatible with the RDF test manifest since we are at an early stage. I asked you to run the tests to find our how your parser deals with the corner cases about which we still discuss.

But following your suggestion, I will bring up the possibility to adapt to the manifest in our next group meeting.

william-vw commented 4 years ago

Update: our current syntax test suite utilizes the RDF test manifest for its Turtle and N3 test cases.

You can find the tests here.

Note that the community group has made some tentative decisions that will likely not sit well with your N3 parser - such as dropping support for the @keywords section.

Feel free to try them out and get back to us. Will be interesting to see what output you get!

RubenVerborgh commented 4 years ago

@william-vw Thanks, that is helpful.

Some problems remain:

https://raw.githubusercontent.com/w3c/N3/master/grammar/tests/N3Tests/manifest.ttl is served with the wrong content type (GitHub pages might help here)
@base is <https://github.com/w3c/N3/tests/>, such that the URLs of tests break. I'd suggest to keep URLs relative.
The manifest description is a blank node rather than <>, such that it cannot be identified as the main node in the document.

I fixed those manually at https://ruben.verborgh.org/tmp/n3/grammar/tests/N3Tests/manifest.ttl and currently have 954 / 1234 tests succeeded. Log below.

n3js-n3-v1.3.5.log

RubenVerborgh commented 4 years ago

In general, I notice that some tests are very large, leading to difficult-to-understand logs. Whereas I see value in testing large corpora of existing files, specification tests usually verify specific syntactic features. Maybe the current N3.js errors can serve as inspiration for what things to test.

RubenVerborgh commented 4 years ago

FYI code in https://github.com/rdfjs/N3.js/tree/feature/n3-spec, failing tests at https://travis-ci.org/rdfjs/N3.js

@rubensworks Also ran into

$ npm run spec-earl-n3

> n3@1.3.5 spec-earl-n3 /Users/ruben/Documents/UGent/N3/N3.js
> rdf-test-suite spec/parser.js https://ruben.verborgh.org/tmp/n3/grammar/tests/N3Tests/manifest.ttl -i '{ "format": "n3" }' -c .rdf-test-suite-cache/ -o earl -p spec/earl-meta.json > spec/earl-n3.ttl

Error: Invalid JavaScript value: 'null'
    at Translator.toRdf (/Users/ruben/Documents/UGent/N3/N3.js/node_modules/rdf-literal/lib/Translator.js:47:15)
    at toRdf (/Users/ruben/Documents/UGent/N3/N3.js/node_modules/rdf-literal/index.js:34:23)
    at module.exports (/Users/ruben/Documents/UGent/N3/N3.js/node_modules/rdf-quad/index.js:9:55)
    at TestSuiteRunner.resultsToEarl (/Users/ruben/Documents/UGent/N3/N3.js/node_modules/rdf-test-suite/lib/TestSuiteRunner.js:205:24)
    at /Users/ruben/Documents/UGent/N3/N3.js/node_modules/rdf-test-suite/bin/Runner.js:73:29
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)

rubensworks commented 4 years ago

@RubenVerborgh Looks like the same problem as https://github.com/rubensworks/rdf-test-suite.js/issues/40, I haven't gotten around to fixing that one yet 😞

william-vw commented 4 years ago

@RubenVerborgh Thanks, I updated the manifest.ttl file. You can find one with proper content type here: https://w3c.github.io/N3/grammar/tests/N3Tests/manifest.ttl

Re our test corpora, this should currently not be seen as a "spec" test suite but rather an effort to include as many test files as possible - our grammar diverges from the prior one since it integrates some productions from Turtle (causing issues with e.g., restrictiveness re escaped characters) and some design decisions from the community group (such as dropping the @keywords section).

william-vw commented 4 years ago

@RubenVerborgh I could be wrong here (only had a brief look) but it looks like N3.js throws errors for the following:

formulas inside lists (e.g., 4color_query.n3, biQ.n3);
literals as subjects (e.g., bell.n3, cryptoP.n3)
N3 path syntax (e.g., fib.n3, alarmE.n3), i.e., using "!" and "^" symbols

Note that your tests include those with approval "Proposed" and "Rejected" - filtering those when reading the manifest.ttl file will rule out some errors as well.

RubenVerborgh commented 4 years ago

@william-vw Thanks! Indeed, the set that trips N3.js seems indeed limited despite the large number of failing tests, but that makes sense given what you wrote above.

I also randomly spotted

:Juno :says  {@forAll :y.  :Mars :fights :y}.

{ :Juno :says  {@forAll :y. :Mars :fights :y } } =>  { :test a :SUCCESS }.

#ends

  Error: Error: Undefined prefix ":" on line 4.

which would be correct for the current Turtle, but apparently not Notation3. But we probably can go into details about each of these when needed.

As far as this issue is concerned, the spec has been added in the feature/n3-spec branch. Now we need to work on compliance in #192.

RubenVerborgh commented 4 years ago

* formulas inside lists (e.g., 4color_query.n3, biQ.n3);

Fixed by https://github.com/rdfjs/N3.js/pull/194.

RubenVerborgh commented 4 years ago

* literals as subjects (e.g., bell.n3, cryptoP.n3)

Fixed by #193.

1024 / 1234 tests passing now

rdfjs / N3.js

N3 tests #191