search

w3c / rdf-tests

Repository for the RDF Tests Community Group
w3c.github.io/rdf-tests
Other
42 stars 23 forks source link

How to run SPARQL Test Suite? #47

Open leipert opened 7 years ago

leipert commented 7 years ago

I really want to contribute new tests for SPARQL REGEX (#46). What is the easiest way to run the suite and produce an implementation report assuming I have an SPARQL endpoint running at http://localhost:8080?

I found kasei/sparql11-protocolvalidator, but I am unfamiliar with PERL (and PERL cgi). Could someone give me a hint?

gkellogg commented 7 years ago

Everyone has their own mechanism for running the tests. Typically, parsing the appropriate test manifest (such as [this one](http://w3c.github.io/rdf-tests/sparql11/data-r2/regex/manifest.ttl, and iterating through each entry. There are some basic instructions for running tests here. Typically, they involve creating a dataset and parsing and running the associated query, where either the result-set or RDF Graph is expected to match that referenced from mf:result.

leipert commented 7 years ago

Thank you for the introduction 👍 This probably means I have to come up with my own mechanism. sigh Is there any test mechanism for test mechanisms? (just kidding) Or I learn a bit PERL ;)

lisp commented 7 years ago

there was a test harness which was introduced by an effort at inria (https://hal.inria.fr/hal-01104252/document), but i do not know its current status. the website (http://sparqlscore.com) is not active. there is a github repository (https://github.com/BorderCloud/TFT)which contains the php code. we had issues with it at the protocol level, but it may serve your needs.

afs commented 7 years ago

Apache Jena provides a command line qtest for running SPARQL-style manifests of tests.

I'm sure other toolkits tat submitted reports provide ways to run the tests.

BorderCloud commented 6 years ago

@lisp Sparqlscore is always active but my time is limited. You can find examples here : https://github.com/BorderCloud/tft-jena-fuseki https://github.com/BorderCloud/tft-virtuoso7-stable https://github.com/BorderCloud/tft-stardog https://github.com/BorderCloud/tft-blazegraph

I will write the doc this week but you can see my script. I started to write a dynamic report (with students).

update: insert new examples

ericprud commented 6 years ago

in SWObjects, you can make test to run the normal tests and TEST_ARGS=--earl make test to emit EARL, c.f. test_SPARQL1.1

rubensworks commented 5 years ago

For reference, here's a tool for executing SPARQL-style test manifests for JavaScript implementations: https://www.npmjs.com/package/rdf-test-suite

namedgraph commented 1 month ago

These links are dead:

I think this issue is still relevant: how do we run the SPARQL test suite?

Also why is the following still the case when we have a standard SPARQL protocol?

Everyone has their own mechanism for running the tests

namedgraph commented 1 month ago

@BorderCloud I don't understand why your approach requires a Docker image per product?

I'm thinking we just need a "test driver" Docker image, so that the actual triplestore is accessed via a SPARQL endpoint URL and the tests are executed over HTTP. Wouldn't that make much more sense?

gkellogg commented 1 month ago

These links are dead:

Where do you still find these links? The first would never have worked with a GitHub repo, the second was obsoleted as part of #84 and #90.

Generally, tests can be run from https://w3c.github.io/rdf-tests/sparql/ by following the manifests.

Exceptions are for the protocol test, which require an endpoint. I spent some time trying to make the manifests contain everything necessary to provision a local server to provide the appropriate responses (see #79), although this hasn't progressed in a while. This would be true for both sparql/sparql11/protocol and sparql/sparql11/http-rdf-update.

I think this issue is still relevant: how do we run the SPARQL test suite?

The top-level README could use some more specific instructions.

Also why is the following still the case when we have a standard SPARQL protocol?

Everyone has their own mechanism for running the tests

I run from https://w3c.github.io/rdf-tests/sparql with a local cache that prevents actual network access. Others run differently. Generally, tests (RDF and SPARQL) should be run as if accessing from GitHub.io, other than the protocol tests which require a runner be provisioned someplace.