Struggling with importing report from automated tools

[Jym77]

So, as a tool vendor, I want to produce reports that can be imported in the WCAG-EM report tool. I'm still struggling to produce a minimal EARL file that gets accepted 😞

The M(N)WE report I'm using:

{
  "@context": {
    "earl": "http://www.w3.org/ns/earl#",
    "WCAG21": "http://www.w3.org/TR/WCAG21/#"
  },
  "@graph": [
    {
      "@type": "earl:Assertion",
      "earl:test": {
        "@id": "WCAG21:non-text-content",
        "@type": ["earl:TestCriterion", "earl:TestRequirement"]
      },
      "earl:subject": {
        "@id": "https://example.org",
        "@type": ["earl:TestSubject"]
      },
      "earl:result": {
        "earl:outcome": {
          "@id": "earl:failed"
        }
      },
      "earl:assertedBy": { "@id": "Alfa" }
    },
    {
      "@type": "earl:Assertion",
      "earl:test": {
        "@id": "WCAG21:name-role-value",
        "@type": ["earl:TestCriterion", "earl:TestRequirement"]
      },
      "earl:subject": {
        "@id": "https://example.org",
        "@type": ["earl:TestSubject"]
      },
        "earl:result": {
          "earl:outcome": {
            "@id": "earl:failed"
          }
        },
      "earl:assertedBy": { "@id": "Alfa" }
    }
  ]
}

So, it seems that in $subjects the URL is stored in the description field, and the id field is just a random ID. But in the assertion, the URL is stored in the id field. Given that this bit of code happens just within this guard: isURL(subjectID), I'm fairly confident that it is expected for subject.id to be indeed an URL 🤔

So, I feel like I'm missing something. Either how to select page and have their URL in the id rather than description, or the other way around how to change my EARL to have the correct id.

---

Changing the comparison in the find from $subject.id === subject.id to $subject.description === subject.id does get me to load the assertions. (and get an import error, but that's another problem…)

[edit: replaced the example EARL file with a working one]

[rvantonisse]

I am not sure, but I think the id like _:subject_1 did also pass the isURL guard. So this needs to be tightened up to exclude json-ld local IRI's as well. Or another guard isIRI preferably to be added. Or...

Matching against (json-ld) id's seemed like a good idea at the time (and adding complexity to retrieve URLS from either key to be set as id). But to identify the subjects by what is required by the tool might be better. Thus matching against title + description, which may be an URl or not. Treating each key as plain (trimmed?) string. This is what the user fills in step 3. And probably is filled by users in other tools as well.

Feels like this requires some more thoughts.

[Jym77]

I am not sure, but I think the id like _:subject_1 did also pass the isURL guard.

No, it doesn't. The isURL guard is very basic and simply tries to call new URL() on its argument, which throws an error on _:subject_1.

[Jym77]

I've finally managed to built a working EARL file 🎉, updated the example in my first message. The file is still somewhat different from what I'm producing automatically, I need to see how important these differences are…

Ultimately the only change I needed to do was on the URL matching:

diff --git a/src/stores/earl/assertionStore/index.js b/src/stores/earl/assertionStore/index.js
index 2a95f48b..a84b7e13 100644
--- a/src/stores/earl/assertionStore/index.js
+++ b/src/stores/earl/assertionStore/index.js
@@ -100,7 +100,7 @@ export async function importAssertions(json) {
       subject.id = subjectID;

       return $subjects.find(($subject) => {
-        return $subject.id === subject.id;
+        return $subject.description === subject.id;
       });
     }

---

Some others points I noticed on the way:

• this test effectively requires the outcome to coded as

  earl:result": {
      "earl:outcome": {
        "@id": "earl:failed"
      }
    },

  (due to the .result.outcome.id chain) and it throw an exception if it's done differently. Looking at ACT implementation reports, it seems that both axe-core and QualWeb produce outcomes as

  earl:result": {
      "earl:outcome": "earl:failed"
    },

  which sounds reasonable and should probably be accepted. (Alfa does provide the outcomes in the correct format, but that's only due to our EARL being excessively verbose…)

• At this point, the outcome of imported assertions is set to "can't tell". I assume this is because automated tools cannot be trusted to provide 100% accurate results, and often only test a subset of a WCAG SC. This means, however, that quite some amount of manual work is still needed after importing to set the results correctly. Should there be a way for tools to say that their result is final? I'm specifically thinking about tools that use WCAG 2 test rules and their fail-on-fail relationship: if the tool says the SC is not satisfied, this should be trusted. Obviously, not all tools have a fail-on-fail relationship, and some could have a pass-on-pass one, so it should be a more generic way of saying "this outcome is definitive, do not replace it with can't tell".
• Similar to the previous point, saying a SC fails on a single page could also make it fail for the entire sample.
• When matching subject and sample URL, the subject URL (subject.id) went through a new URL(…).href loop (through getURL) while the sample URL (was $subject.id, changed to $subject.description) doesn't. The new URL loop effectively adds a trailing slash to URL with no path (e.g. turns https://example.org into https://example.org/). This lead to a very confusing moment where both my EARL and my sample were using https://example.org (no trailing slash) and I realised I had to add one to the sample URL to get the match working 🙈 I guess the same kind of normalisation should happen on both sides.

---

Don't get me wrong, I'm happy to help improve things (within the limited time I have…) Getting automated tools results directly in the report would be awesome… I don't know the project much, but let me know if/how I can help.

[Jym77]

One more (recording all findings in this issue…) At this point, matchCriterion essentially expects the SC to be passed as WCAG21:name-role-value (and splits on the colon). This breaks badly if the SC is passed as https://www.w3.org/TR/WCAG/#name-role-value since it then tries to match //www.w3.org/TR/WCAG/#name-role-value with SC names. I feel the URL does also make sense and should likely be accepted.

[rvantonisse]

All imported // opened json data is expected to be json-ld. As such it is treated with the json-ld api first, given the context of this app. So essentially all following json data is treated knowing it is formated in a way that certain keys and value formats are expected before working with it. (Typing this in github ios app, will point to some essential files later) Having basic knowledge of json-ld and its api is essential when working with earl and this app's data.

[Jym77]

When matching criterion, the test is expected to be part of at most one WCAG SC (getting test.isPartOf.id). This is not necessarily true, especially some ACT rules like Text has minimum contrast map to several SCs (1.4.3 and 1.4.6 in this case). For such a rule, Alfa and QualWeb do provide test.isPartOf as an array.

This effectively causes the import tool to completely drop the full result (for all corresponding SC) since it cannot find test.isPartOf.id.

[rvantonisse]

When matching criterion, the test is expected to be part of at most one WCAG SC (getting test.isPartOf.id).

This is not necessarily true, especially some ACT rules like Text has minimum contrast map to several SCs (1.4.3 and 1.4.6 in this case).

For such a rule, Alfa and QualWeb do provide test.isPartOf as an array.

This effectively causes the import tool to completely drop the full result (for all corresponding SC) since it cannot find test.isPartOf.id.

This is a good point. Simply basing on json-ld, array values should also be accepted. This means however, that these kind of test results are inserted at multiple wcag criteria results. matching on result.pointer (or something similar) should be added.

[Jym77]

When matching criterion, the test is expected to be part of at most one WCAG SC (getting test.isPartOf.id). This is not necessarily true, especially some ACT rules like Text has minimum contrast map to several SCs (1.4.3 and 1.4.6 in this case). For such a rule, Alfa and QualWeb do provide test.isPartOf as an array. This effectively causes the import tool to completely drop the full result (for all corresponding SC) since it cannot find test.isPartOf.id.

This is a good point. Simply basing on json-ld, array values should also be accepted. This means however, that these kind of test results are inserted at multiple wcag criteria results. matching on result.pointer (or something similar) should be added.

Yes. I've (locally) patched that by having matchCriterion (and subsequently findMatch) return an array instead of a single value, that I can then traverse with a for (const matchedResult of matchedResults) when preparing imports and building _importable.