Closed darthcav closed 10 years ago
nitedog
commented
10 years ago Partial - I think we need to alert the reader that they need to pay attention to the fact that some tool vendors might claim to support a testing mode but only do that very scarcely. However, I don't think we can define any thresholds or ways for objectively measuring the level at which they carry out particular types of testing.
Added new section 2.2.3 including the comments.
ysmartin
commented
10 years ago From my understanding a useful feature would be having documentation on the specific tests (either coming from WCAG techniques or elswhere) applied by the tool. This is broader than saying how many automatic, how many semi and how many manual; and at the same time it is not such a heavy assertion. Edit: this is related as well to #2, which asks for reliability figures. My suggestion is not so hard: just document how your tool evaluate each checkpoint (without stating anything on whether that is a reliable fashion of evaluating it or not)
Reference
http://lists.w3.org/Archives/Public/public-wai-ert-tools/2014Aug/0004
Original Comment
"Test modes: automatic, semiautomatic and manual" : The quantity for each is relevant: how many techniques or failures are fully automatic tested, how many tests have a semi-automatic mode and how many tests are covered in the manual mode? It's very hard to compare tools at the moment as some claim that a technique or failure is tested, but in fact only a few aspects are implemented, or there is only a selector added.
When we had to make the decision which tool to use, we made Excel sheets with all the accessibility tools on the market and their claims on which WCAG2 techniques are covered. Comparison was hard as tools aren't explicit if a technique or failure is fully implemented.