Add JSON Schema for test grammar

[halindrome]

The test grammar needs a JSON Schema so that tests, assertion objects, and condition objects can be checked during test build.

We also need a JSON-LD context definition for the grammar. @gkellogg is that something you might be up for?

[gkellogg]

Sure, I can help with the context definition.

[gkellogg]

The existing test examples use https://www.w3.org/ns/JSONtest-v1.jsonld, which doesn't seem appropriate, as it's not really a namespace document. Given that these are part of the annotation work, probably should be in the Web Annotation timestamped directory, something like https://www.w3.org/2015/web-annotation/tests/context.jsonld? @iherman or @shepazu would probably best know the most appropriate location. Alternatively, as it's part of the web-platform tests, someplace publicly available within w3c-test.org could work.

Remember, that the file will need to resolve as application/ld+json with CORS headers set.

[gkellogg]

Note that, tests such as example2.test mix the test description with the assertions, which are JSON Schema, rather than JSON-LD. This may mean needing to invent a JSON-LD context to describe a JSON Schema, which may not really be possible.

Alternatively, the test could reference the assertions as different files which are simply evaluated as JSON Schema documents, not JSON-LD. Or, the schema could be encoded as literals, which are parsed into JSON Schema.

[halindrome]

The CONTRIBUTING.md file shows the structure of the .test files... and yes, it permits the JSON Schema inline.

I didn't realize that JSON Schema could not be expressed in JSON-LD. Is it because of the complex nesting? The terms are all definable, obviously.

Is there a way to treat the assertionObject or comparisonObject as a blob?

On Wed, Jun 15, 2016 at 1:46 PM, Gregg Kellogg notifications@github.com wrote:

Note that, tests such as example2.test https://github.com/Spec-Ops/web-platform-tests/blob/master/annotation-model/examples/example2.test mix the test description with the assertions, which are JSON Schema, rather than JSON-LD. This may mean needing to invent a JSON-LD context to describe a JSON Schema, which may not really be possible.

Alternatively, the test could reference the assertions as different files which are simply evaluated as JSON Schema documents, not JSON-LD. Or, the schema could be encoded as literals, which are parsed into JSON Schema.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Spec-Ops/web-platform-tests/issues/5#issuecomment-226282673, or mute the thread https://github.com/notifications/unsubscribe/AAfx8C2vsxGJsEf1BySa_HOD7F5RilTiks5qMEiZgaJpZM4I1RhH .

Shane McCarron halindrome@gmail.com

[halindrome]

The existing test examples use https://www.w3.org/ns/JSONtest-v1.jsonld, which doesn't seem appropriate, as it's not really a namespace document. Given that these are part of the annotation work, probably should be in the Web Annotation timestamped directory, something like https://www.w3.org/2015/web-annotation/tests/context.jsonld? @iherman or @shepazu would probably best know the most appropriate location. Alternatively, as it's part of the web-platform tests, someplace publicly available within w3c-test.org could work.

Good point. I picked something generic because I thought (and think) that this is not going to be specific to annotation. So I don't care where it is... it just needs to work. I wonder if w3c-test.org under the tools directory would work? We have substantial control of the headers that are emitted in that environment.

[iherman]

@gkellogg :

The existing test examples use https://www.w3.org/ns/JSONtest-v1.jsonld, which doesn't seem appropriate, as it's not really a namespace document. Given that these are part of the annotation work, probably should be in the Web Annotation timestamped directory, something like https://www.w3.org/2015/web-annotation/tests/context.jsonld? @iherman or @shepazu would probably best know the most appropriate location. Alternatively, as it's part of the web-platform tests, someplace publicly available within w3c-test.org could work.

Indeed, /ns is probably not the appropriate place. If you guys prefer not to host it at w3c-test.org (although that may be the most appropriate place), the best place on W3C would probably something like http://www.w3.org/annotation/v1/tests/... or http://www.w3.org/annotation/2016/tests/... The latter is more in line with the usual W3C approach of adding dates to these things, 'v1' is probably a better mnemonic for later. I am mildly in favour of 'v1' if we go down that route.

[halindrome]

I am happy to put it on w3c-test.org. I will do a test on my local instance with setting the headers as we need. It should just work. Unfortunately, the URL path space on that site is going to be dicey... we will end up with w3c-test.org/annotation-model/jsontest-v1.jsonld or something like that - is that okay?

On Thu, Jun 16, 2016 at 3:28 AM, Ivan Herman notifications@github.com wrote:

@gkellogg https://github.com/gkellogg :

The existing test examples use https://www.w3.org/ns/JSONtest-v1.jsonld, which doesn't seem appropriate, as it's not really a namespace document. Given that these are part of the annotation work, probably should be in the Web Annotation timestamped directory, something like https://www.w3.org/2015/web-annotation/tests/context.jsonld? @iherman https://github.com/iherman or @shepazu https://github.com/shepazu would probably best know the most appropriate location. Alternatively, as it's part of the web-platform tests, someplace publicly available within w3c-test.org could work.

Indeed, /ns is probably not the appropriate place. If you guys prefer not to host it at w3c-test.org (although that may be the most appropriate place), the best place on W3C would probably something like http://www.w3.org/annotation/v1/tests/... or http://www.w3.org/annotation/2016/tests/... The latter is more in line with the usual W3C approach of adding dates to these things, 'v1' is probably a better mnemonic for later. I am mildly in favour of 'v1' if we go down that route.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Spec-Ops/web-platform-tests/issues/5#issuecomment-226420955, or mute the thread https://github.com/notifications/unsubscribe/AAfx8HjzIROAYKD-gU7RxYJag0LPceuaks5qMQkmgaJpZM4I1RhH .

Shane McCarron halindrome@gmail.com

[iherman]

On 16 Jun 2016, at 13:33, Shane McCarron notifications@github.com wrote:

I am happy to put it on w3c-test.org. I will do a test on my local instance with setting the headers as we need. It should just work. Unfortunately, the URL path space on that site is going to be dicey... we will end up with w3c-test.org/annotation-model/jsontest-v1.jsonld or something like that - is that okay?

Works for me…

I.

On Thu, Jun 16, 2016 at 3:28 AM, Ivan Herman notifications@github.com wrote:

@gkellogg https://github.com/gkellogg :

The existing test examples use https://www.w3.org/ns/JSONtest-v1.jsonld, which doesn't seem appropriate, as it's not really a namespace document. Given that these are part of the annotation work, probably should be in the Web Annotation timestamped directory, something like https://www.w3.org/2015/web-annotation/tests/context.jsonld? @iherman https://github.com/iherman or @shepazu https://github.com/shepazu would probably best know the most appropriate location. Alternatively, as it's part of the web-platform tests, someplace publicly available within w3c-test.org could work.

Indeed, /ns is probably not the appropriate place. If you guys prefer not to host it at w3c-test.org (although that may be the most appropriate place), the best place on W3C would probably something like http://www.w3.org/annotation/v1/tests/... or http://www.w3.org/annotation/2016/tests/... The latter is more in line with the usual W3C approach of adding dates to these things, 'v1' is probably a better mnemonic for later. I am mildly in favour of 'v1' if we go down that route.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Spec-Ops/web-platform-tests/issues/5#issuecomment-226420955, or mute the thread https://github.com/notifications/unsubscribe/AAfx8HjzIROAYKD-gU7RxYJag0LPceuaks5qMQkmgaJpZM4I1RhH .

Shane McCarron halindrome@gmail.com — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Spec-Ops/web-platform-tests/issues/5#issuecomment-226460057, or mute the thread https://github.com/notifications/unsubscribe/AAfyExEtWUV5CHoVrLkti0P-N-ILvy6Eks5qMTSdgaJpZM4I1RhH.

---

Ivan Herman, W3C Digital Publishing Lead Home: http://www.w3.org/People/Ivan/ mobile: +31-641044153 ORCID ID: http://orcid.org/0000-0003-0782-2704

[gkellogg]

A lot of what we need to do depends on the purpose of interpreting the test files as JSON-LD. Unless you're expecting to round-trip a file, it's probably not necessary to interpret everything as JSON-LD. The context I created in f241c480b42ddfef3167df5b86b20120dbb055c4 is sufficient to retrieve basic information about the test as RDF, which would be useful in using it to generate information for an implementation report. In fact, the assertions term isn't really necessary.

A JSON-LD processor will ignore properties that don't expand to an absolute IRI, so for example, https://github.com/Spec-Ops/web-platform-tests/blob/master/annotation-model/examples/example2.test would expand to something like the following Turtle:

@prefix dc: <http://purl.org/dc/elements/1.1/> .
@prefix test: <http://www.w3.org/2006/03/test-description#> .

[
  dc:name "A test that has an 'or' clause";
  dc:description "This is a complex test that uses or-ing among a list of assertions.";
  test:specificationReference <https://www.w3.org/TR/annotation-model/#model>;
]

The test could be improved by adding a @type and @id (or type and id with suitable aliases). For example, simply using {"@id": ""} would be sufficient to use the location of the test as the subject for the test. At a minimum adding "@type": "test:TestCase" would identify these as being tests, but we might subclass this to create a more specific type of annotation/test schema based test.

As for location, where the commit places the context should allow it to be used as https://web-platform.test/annotation-model/context.jsonld, using the localhost-aliases for the web platform. A processor running where these aliases are in place should retrieve it properly (modulo appropriate CORS headers). Of course, when properly synced, this should also work as https://w3c-test.org/annotation-model/context.jsonld. If there's a desire to use this for more test platform tests, it might simply be https://w3c-test.org/context.jsonld.

An example test might be the following:

{
  "@context": "https://web-platform.test/annotation-model/context.jsonld",
  "@id": "",
  "@type": "test:TestCase",
   "name": "A test that has an 'or' clause",
   "description": "This is a complex test that uses or-ing among a list of assertions.",
   "ref": "https://www.w3.org/TR/annotation-model/#model",
   "testType": "manual",
   "assertions":
      { "title": "Condition Object",
        "description": "A pseudo-test that will get a result from the aggregate of its children",
        "assertionType": "must",
        "expectedResult": "valid",
        "errorMessage": "Error: None of the various options were present",
        "compareWith": "or",
        "assertions": [
        {
          "$schema": "http://json-schema.org/draft-04/schema#",
          "title": "'The Annotation must have 1 or more @context values' (Section 3.1)",
          "assertionType": "must",
          "expectedResult": "valid",
          "errorMessage": "Error: Annotation does not have an @context property.",
          "type": "object",
          "properties": {
            "@context": {}
          },
          "required": ["@context"]
        },
        {
          "$schema": "http://json-schema.org/draft-04/schema#",
          "title": "'An Annotation should have exactly 1 id' (Section 3.1)",
          "assertionType": "should",
          "expectedResult": "valid",
          "errorMessage": "Warning: The Annotation is not identified using the id key (Section 3.1).",
          "type": "object",
          "properties": {
            "id": {}
          },
          "required": ["id"]
        }
        ]
      }
}

[gkellogg]

@halindrome and I did discuss the difference of a test and sub-tests, with each assertion treated as a sub-test. Not really a way to do this with a straight JSON-LD transformation. We could, however, imagine some custom code which creates a test for each assertion by using the assertion title (or location, if a reference) to create a URI fragment to apply to the test file location. In this case, there's not a standardized way to turn a test file into EARL test descriptions, but an algorithm. Otherwise, we simply treat the collection of assertions as an atomic part of the test, which is either pass/fail based on passing all assertions or not.