search

solid-contrib / test-suite

An automated test of Solid specification technical compliance
MIT License
23 stars 10 forks source link

Objective of the test suite #84

Open kjetilk opened 4 years ago

kjetilk commented 4 years ago

We need to decide whether the test suite should align closely with the spec, or with NSS as of now. I started to do the latter, but believe now that the former is the right approach.

kjetilk commented 4 years ago

Unless someone speaks up, I will adapt the test suite to conform to the spec going forward.

timbl commented 4 years ago

If you find that they are different, then you have to figure out whether it is a bug with NSS or a bug with the spec. Don't assume one or the other. If it isn't obvious which is wrong, then raise an issue with both and escalate it

kjetilk commented 4 years ago

Yes, indeed, that is the idea. We are sticking to NSS as the guideline, but oftentimes, it has been underdefined. For example, you can ask NSS to create a Non-RDF resource but at the same time declare that it is a container with a Link header. NSS will do that without complaint, but now we say that it is an internally inconsistent situation. People are unlikely do have ever tried to do it, because it is a silly thing to do, but we shouldn't have those kinds of inconsistencies, and so we have quite a lot of tests that NSS is failing. Those doesn't change the way Solid has de facto been behaving, but it rectifies the ways that have been underdefined.

kjetilk commented 4 years ago

This comment is an example of one of the more ambiguous issues I have encountered: https://github.com/solid/specification/issues/40#issuecomment-585154494

csarven commented 4 years ago

Related: as an interim solution, should the NSS repo state that it reflects solid-spec (as opposed to solid/specification) .. and that the current solid/test-suite may give different results?