LJWatson
commented
5 years ago This is going to be relevant when testing websites too. How much of the site should be tested before someone can claim conformance? Currently I think this is only ever done by counting numbers of pages. But I don't think that has to be the only way to do it.>
It's only possible to claim conformance at the document/page level in WCAG, though a claim may encompass multiple pages. The practice of using a representative sample of pages to claim website level conformance is recognition of the fact that document/page level conformance for an entire website is impractical to the point of being impossible.
The trouble with representative samples, as with any form of partial coverage, is what happens when the really important stuff is in the part of the site/page that isn't covered? We all know the expectation is that issues will be extrapolated from the test sample and resolved everywhere, but the reality is that this just doesn't happen.
EmmaJP
MakotoUeki
A challange in WCAG 1 and 2 is that it's impossible to say anything about conformance unless a web page is fully tested. Because you don't always have a complete picture, it is difficult to say anything about the accessibility. It'd be very helpful if in addition to allowing pages to conform if not 100% of the content passes the requirements, to also allow conformance claims of pages that aren't 100% tested.
This is going to be relevant when testing websites too. How much of the site should be tested before someone can claim conformance? Currently I think this is only ever done by counting numbers of pages. But I don't think that has to be the only way to do it.
This could in theory work if applicability of a requirement could be made programatically determinable. For instance, if a requirement is applicable to all img elements, and a page has 100 of them, with 90 img elements tested, you have 90% test coverage for that requirements. WCAG 2 doesn't have this concent of applicability, making a measurement like this impossible.
If we allow incidental issues in conformance of Silver, it makes sense to also allow test coverage to be lower than 100%. By writing Silver in such a way that it can be used to measure test coverage, it becomes possible to maintain conformance over time. Rather than have the conformance claim get outdated the moment a page changes, conformance claim can stay relevant as long as the test target didn't change too much. Even better, you can programatically determine the moment the conformance claim becomes outdated, as that will be the moment the test coverage drops below a certain level.