What do you do, if you have a cold and must stay at home? You write tests for ARC2! :trollface:
W3C test suite
There is a test suite for SPARQL 1.1 compliance provided by the W3C here: https://www.w3.org/2009/sparql/docs/tests/. It contains many tests for different types of queries. They range from simple SELECT queries over function-usage to graph management.
Goal and motivation
I want to know the current state of SPARQL 1.0/1.1 support of ARC2. Instead of using custom queries, i decided to use the official test suite. Their tests showed that ARC2 lacks major parts of SPARQL 1.0/1.1. Please do not understand that as blaming, but as a first step to define a basement for further development. With these tests, further changes on the query engine can be made without worrying about adding bugs or change expected behavior.
Tested ARC2 component
These new tests check the result from ARC2_Store::query for a given query.
Coverage
I picked mostly cases, which cover major functionality. In the following related folders from the test suite:
aggregates
construct
delete
drop
exists
grouping
move
syntax-update-1
Additionally, there a some cases which have exotic kinds of configuration (manifest.ttl). A few of those are also part of the list.
7 folders are covered, 22 folders are not covered yet.
Important: How to interpret the new tests
Don't get fooled, there are 2 different kinds of the tests:
First kind of tests actually test nothing, because i manually checked if the functionality is available. That is the case for instance for the MOVE command. Based on this wiki page. Nevertheless, i ran some tests, but they approved the wiki page information. I marked those tests as skipped on purpose, to signal lacks in the SPARQL support. If all these would be successful, one could get the impression, ARC2 has no lacks in SPARQL support.
The other kind of tests check the functionality. They setup the test environment based on a given manifest.ttl, load data etc. and run a W3C-provided query. A test is approved, if the expected result (given by the W3C) matches the actual result (returned from ARC2).
Every skipped test provides meaningful information. For instance Tests\sparql11\AggregatesTest::test_agg_avg_01 outputs:
Test skipped, because of rounding bug in AVG function.
See https://github.com/semsol/arc2/issues/99
This way one can see the limitations of ARC2, sometimes also the reason. If one decides to extend/fix ARC2 SPARQL support, they can take the related test and change it.
Current output in PHPUnit
New PHPUnit output below. To see a full overview, you can look here.
.SS.SSSSSSS.SSS..S..SS.S....S.....SSSSSSSSSSSSSSSSSSSSSSS.SSSSS 63 / 171 ( 36%)
..SS...SSSSSSS..SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS......... 126 / 171 ( 73%)
.S.S......................................... 171 / 171 (100%)
Time: 32 seconds, Memory: 12.00MB
There were 94 skipped tests:
1) Tests\sparql11\AggregatesTest::test_agg_avg_01
Test skipped, because of rounding bug in AVG function.
See https://github.com/semsol/arc2/issues/99
/home/travis/build/k00ni/arc2/tests/sparql11/AggregatesTest.php:29
/home/travis/build/k00ni/arc2/vendor/phpunit/phpunit/phpunit:52
2) Tests\sparql11\AggregatesTest::test_agg_avg_02
This kind of query is currently not supported. ARC2_Store::query returns 0 for query:
PREFIX : <http://www.example.org/>
SELECT ?s (AVG(?o) AS ?avg)
FROM <http://arc/data/> WHERE { ?s ?p ?o
}
GROUP BY ?s
HAVING (AVG(?o) <= 2.0)
/home/travis/build/k00ni/arc2/tests/sparql11/AggregatesTest.php:41
/home/travis/build/k00ni/arc2/vendor/phpunit/phpunit/phpunit:52
[...]
Further changes
Besides these new files in tests/sparql11, i changed the following files:
.travis.yml - added PHP 7.2 as further backend
composer.json - Changed some require-dev entries to fix failing tests in Travis.
phpunit.xml - Separated test suites, in case, one wants to run tests without compliance tests.
tests/ARC2_TestCase.php - Small refinements and further comments.
tests/bootstrap.php - Explicit force error_reporting(E_ALL)
Related issues
During this process, i opened the following issues:
What do you do, if you have a cold and must stay at home? You write tests for ARC2! :trollface:
W3C test suite
There is a test suite for SPARQL 1.1 compliance provided by the W3C here: https://www.w3.org/2009/sparql/docs/tests/. It contains many tests for different types of queries. They range from simple SELECT queries over function-usage to graph management.
Goal and motivation
I want to know the current state of SPARQL 1.0/1.1 support of ARC2. Instead of using custom queries, i decided to use the official test suite. Their tests showed that ARC2 lacks major parts of SPARQL 1.0/1.1. Please do not understand that as blaming, but as a first step to define a basement for further development. With these tests, further changes on the query engine can be made without worrying about adding bugs or change expected behavior.
Tested ARC2 component
These new tests check the result from
ARC2_Store::query
for a given query.Coverage
I picked mostly cases, which cover major functionality. In the following related folders from the test suite:
Additionally, there a some cases which have exotic kinds of configuration (
manifest.ttl
). A few of those are also part of the list.7 folders are covered, 22 folders are not covered yet.
Important: How to interpret the new tests
Don't get fooled, there are 2 different kinds of the tests:
manifest.ttl
, load data etc. and run a W3C-provided query. A test is approved, if the expected result (given by the W3C) matches the actual result (returned from ARC2).Every skipped test provides meaningful information. For instance
Tests\sparql11\AggregatesTest::test_agg_avg_01
outputs:This way one can see the limitations of ARC2, sometimes also the reason. If one decides to extend/fix ARC2 SPARQL support, they can take the related test and change it.
Current output in PHPUnit
New PHPUnit output below. To see a full overview, you can look here.
Further changes
Besides these new files in
tests/sparql11
, i changed the following files:Related issues
During this process, i opened the following issues:
99
100