Open Jym77 opened 2 years ago
I am not sure, but I think the id like _:subject_1
did also pass the isURL guard. So this needs to be tightened up to exclude json-ld local IRI's as well. Or another guard isIRI preferably to be added. Or...
Matching against (json-ld) id's seemed like a good idea at the time (and adding complexity to retrieve URLS from either key to be set as id). But to identify the subjects by what is required by the tool might be better. Thus matching against title + description, which may be an URl or not. Treating each key as plain (trimmed?) string. This is what the user fills in step 3. And probably is filled by users in other tools as well.
Feels like this requires some more thoughts.
I am not sure, but I think the id like
_:subject_1
did also pass the isURL guard.
No, it doesn't. The isURL
guard is very basic and simply tries to call new URL()
on its argument, which throws an error on _:subject_1
.
I've finally managed to built a working EARL file 🎉, updated the example in my first message. The file is still somewhat different from what I'm producing automatically, I need to see how important these differences are…
Ultimately the only change I needed to do was on the URL matching:
diff --git a/src/stores/earl/assertionStore/index.js b/src/stores/earl/assertionStore/index.js
index 2a95f48b..a84b7e13 100644
--- a/src/stores/earl/assertionStore/index.js
+++ b/src/stores/earl/assertionStore/index.js
@@ -100,7 +100,7 @@ export async function importAssertions(json) {
subject.id = subjectID;
return $subjects.find(($subject) => {
- return $subject.id === subject.id;
+ return $subject.description === subject.id;
});
}
Some others points I noticed on the way:
earl:result": {
"earl:outcome": {
"@id": "earl:failed"
}
},
(due to the .result.outcome.id
chain) and it throw an exception if it's done differently. Looking at ACT implementation reports, it seems that both axe-core and QualWeb produce outcomes as
earl:result": {
"earl:outcome": "earl:failed"
},
which sounds reasonable and should probably be accepted. (Alfa does provide the outcomes in the correct format, but that's only due to our EARL being excessively verbose…)
subject.id
) went through a new URL(…).href
loop (through getURL
) while the sample URL (was $subject.id
, changed to $subject.description
) doesn't. The new URL
loop effectively adds a trailing slash to URL with no path (e.g. turns https://example.org
into https://example.org/
). This lead to a very confusing moment where both my EARL and my sample were using https://example.org
(no trailing slash) and I realised I had to add one to the sample URL to get the match working 🙈 I guess the same kind of normalisation should happen on both sides.Don't get me wrong, I'm happy to help improve things (within the limited time I have…) Getting automated tools results directly in the report would be awesome… I don't know the project much, but let me know if/how I can help.
One more (recording all findings in this issue…)
At this point, matchCriterion
essentially expects the SC to be passed as WCAG21:name-role-value
(and splits on the colon). This breaks badly if the SC is passed as https://www.w3.org/TR/WCAG/#name-role-value
since it then tries to match //www.w3.org/TR/WCAG/#name-role-value
with SC names. I feel the URL does also make sense and should likely be accepted.
All imported // opened json data is expected to be json-ld. As such it is treated with the json-ld api first, given the context of this app. So essentially all following json data is treated knowing it is formated in a way that certain keys and value formats are expected before working with it. (Typing this in github ios app, will point to some essential files later) Having basic knowledge of json-ld and its api is essential when working with earl and this app's data.
When matching criterion, the test is expected to be part of at most one WCAG SC (getting test.isPartOf.id
).
This is not necessarily true, especially some ACT rules like Text has minimum contrast map to several SCs (1.4.3 and 1.4.6 in this case).
For such a rule, Alfa and QualWeb do provide test.isPartOf
as an array.
This effectively causes the import tool to completely drop the full result (for all corresponding SC) since it cannot find test.isPartOf.id
.
When matching criterion, the test is expected to be part of at most one WCAG SC (getting
test.isPartOf.id
).This is not necessarily true, especially some ACT rules like Text has minimum contrast map to several SCs (1.4.3 and 1.4.6 in this case).
For such a rule, Alfa and QualWeb do provide
test.isPartOf
as an array.This effectively causes the import tool to completely drop the full result (for all corresponding SC) since it cannot find
test.isPartOf.id
.
This is a good point. Simply basing on json-ld, array values should also be accepted. This means however, that these kind of test results are inserted at multiple wcag criteria results. matching on result.pointer (or something similar) should be added.
When matching criterion, the test is expected to be part of at most one WCAG SC (getting
test.isPartOf.id
). This is not necessarily true, especially some ACT rules like Text has minimum contrast map to several SCs (1.4.3 and 1.4.6 in this case). For such a rule, Alfa and QualWeb do providetest.isPartOf
as an array. This effectively causes the import tool to completely drop the full result (for all corresponding SC) since it cannot findtest.isPartOf.id
.This is a good point. Simply basing on json-ld, array values should also be accepted. This means however, that these kind of test results are inserted at multiple wcag criteria results. matching on result.pointer (or something similar) should be added.
Yes. I've (locally) patched that by having matchCriterion
(and subsequently findMatch
) return an array instead of a single value, that I can then traverse with a for (const matchedResult of matchedResults)
when preparing imports and building _importable
.
So, as a tool vendor, I want to produce reports that can be imported in the WCAG-EM report tool. I'm still struggling to produce a minimal EARL file that gets accepted 😞
The M(N)WE report I'm using:
So, it seems that in
$subjects
the URL is stored in thedescription
field, and theid
field is just a random ID. But in the assertion, the URL is stored in theid
field. Given that this bit of code happens just within this guard:isURL(subjectID)
, I'm fairly confident that it is expected forsubject.id
to be indeed an URL 🤔So, I feel like I'm missing something. Either how to select page and have their URL in the
id
rather thandescription
, or the other way around how to change my EARL to have the correctid
.Changing the comparison in the
find
from$subject.id === subject.id
to$subject.description === subject.id
does get me to load the assertions. (and get an import error, but that's another problem…)[edit: replaced the example EARL file with a working one]