Input format for test runners

kasperisager commented 5 years ago

This is a proposal for adding context to the generated test cases - https://auto-wcag.github.io/auto-wcag/auto-wcag-testcases/testcases.json. Idea being to embrace TCDL, to express test cases.

{
  "@context": {
    "earl": "http://www.w3.org/ns/earl#",
    "xlink": "http://www.w3.org/1999/xlink/",
    "tcdl": "http://bentoweb.org/refs/TCDL2.0/",
    "a11y-testcases": {
      "@id": "tcdl:testCaseDescription",
      "@container": "@list"
    },
    "rulePage": {
      "@id": "tcdl:rule",
      "@type": "xlink:href"
    },
    "expected": {
      "@id": "tcdl:expectedResult",
      "@type": "@vocab"
    },
    "passed": {
      "@id": "earl:passed"
    }
  },
  "name": "auto-wcag test cases",
  "website": "https://auto-wcag.github.io/auto-wcag",
  "license": "https://auto-wcag.github.io/auto-wcag/license.html",
  "description": "Test Cases of auto-wcag rules",
  "a11y-testcases": [
    {
      "url": "https://auto-wcag.github.io/auto-wcag/auto-wcag-testcases/assets/SC1-1-1-image-has-name_passed_example_1.html",
      "relativeUrl": "assets/SC1-1-1-image-has-name_passed_example_1.html",
      "successCriteria": [
        "WCAG2:non-text-content"
      ],
      "expected": "passed",
      "ruleId": "SC1-1-1-image-has-name",
      "rulePage": "https://auto-wcag.github.io/auto-wcag/rules/SC1-1-1-image-has-name.html"
    },
    {
      "url": "https://auto-wcag.github.io/auto-wcag/auto-wcag-testcases/assets/SC1-1-1-image-has-name_passed_example_2.html",
      "relativeUrl": "assets/SC1-1-1-image-has-name_passed_example_2.html",
      "successCriteria": [
        "WCAG2:non-text-content"
      ],
      "expected": "passed",
      "ruleId": "SC1-1-1-image-has-name",
      "rulePage": "https://auto-wcag.github.io/auto-wcag/rules/SC1-1-1-image-has-name.html"
    }
  ]
}

dd8 commented 5 years ago

TCDL is defined in XML using an XML schema with lots of schema constraints on allowed data types for elements, and whether elements are optional / required / occurs a specified number of times.

I'm guessing the data types can be expressed in JSON-LD, but what about signalling elements are optional / required / occur a specified number of times ?

These seem quite important constraints for a data interchange format. Would JSON Schema help here or is too immature? See https://github.com/json-schema-org/json-schema-spec/issues/612 for background.

There will be interop problems if there's no way of validating the TCDL is correct before submitting it (could write a tool to validate - but this means exchanging code as well as data)

dd8 commented 5 years ago

Another important consideration on the input format is how easy it is to consume with different language types:

compiled + statically typed with strong typing (Java, C++, C#, Objective-C, Swift, Rust, Go)
compiled + dynamically typed with strong typing (Python)
interpreted + statically typed (e.g. OCaml , UCSD Pascal)
interpreted + dynamically typed with strong typing (Ruby - 9+"9" illegal)
interpreted + dynamically typed with weak typing (JavaScript - 9+"9" legal)

Dynamic, weakly typed input formats are much easier to consume with dynamic, weakly typed languages than with compiled, statically typed languages.

jeeyyy commented 5 years ago

Reference:

Test Cases: https://auto-wcag.github.io/auto-wcag/auto-wcag-testcases/testcases.json

Earl Outputs: https://github.com/w3c/earl/tree/master/earl-reports

act-rules / act-rules.github.io

Input format for test runners #333