search

solid-contrib / test-suite

An automated test of Solid specification technical compliance
MIT License
23 stars 10 forks source link

Reporting tests results by solid server #109

Open bourgeoa opened 3 years ago

bourgeoa commented 3 years ago

The actual presentation is made by tests group. This has a drawback when test-suite is changing. That will not be frequent when the test-suite will be stable.

Actually I don't know how to report the results for NSS passing all tests on today version of the tests.

There are different possibilities :

edwardsph commented 3 years ago

I was thinking that we will need to wrap a process around the various test suites that builds a spec conformance document which is published, e.g. https://dvcs.w3.org/hg/ldpwg/raw-file/default/tests/reports/ldp.html. We need to look at how to move from a simple pass/fail count to determining whether the feature is required or optional so we don't report an implementation as failing if it doesn't implement an optional feature. If there is a process that combines these reports then there are 2 paths to getting results included:

  1. The suite runs tests on all servers after a change in the test suite (or perhaps runs a subset of the tests)
  2. An implementor requests a re-test as a result of a change to the implementation

To move forward on this we will also need a better understanding of how tests cover different aspects of the specification so we can also report on the test coverage of the specification.