Open Jym77 opened 1 year ago
This touches the more global question of "how do we test definitions?"
In that case, we have 50 rules who use "visible" and this would maybe need 3-4 cases to fully test the various corner cases (maybe more). It clearly doesn't make sense to add these to every rule.
Another way could be to spread examples across rules, adding 1 to some of the rules, another one to some other rules. However, this means that tools who do not correctly support the definition would randomly fail one rule but not another, which seems a bit arbitrarily. It also makes it harder for us to ensure that the definition is correctly covered by all the test cases, since the tests cases would be spread in the rules instead of being gathered in one place. This can create a big extra maintenance cost.
We did try to add some examples of the definition for the big definitions. These are intended mostly for tool developers. However, these are not automatically testable, and as far as I know they are not shown on the WAI website either (and not really maintained…)
Some ideas and questions:
Let's try to have "rules" for testing definition.
Put the rules in a separate directory (not __rules
).
Put the test cases in a separate JSON file.
We (ACT rules) will not publish any kind of result of implementation reports but that would be a resource for implementers that we provide.
Maybe we can use the web platform tests to store the tests that make sense at that level, for example they seem to already have some accessible name tests, maybe not to the full extend but we should be able to provide more test cases (and everybody benefits…) Need to investigate more in that direction. OTOH, for definitions like "visible" , WPT might not be the best place as UA typically don't care about that directly.
I'm just now finding this conversation and the associated PR, I think having this notion of testing definitions might be very helpful for our current formulation of subjective applicability where most of the subjectivity is held within definitions.
My only counter to this is that we might run into the situation where it feels like any rule with a subjectivity feels like two rules:
I'm not sure if this would cause problems or create additional overhead maintenance, but is probably worth a discussion.
I think we only need to test definitions separately when they are broadly used in other rules. So for now mostly the "core concepts" of visibility, and inclusion in the accessibility tree (role and name are tested by the Web Platform Tests and we should direct implementers to these). For definitions that are mostly used in one rule, it makes sense to have all the tests as part of the rule.
A problem I've been considering while trying to improve the visibility tests in Alfa…
As per our definition of visibility, content that can be scrolled into viewport is considered visible. I guess this was mostly added for vertical scrolling, which is the most frequent case, but it also covers horizontal scrolling:
Here, the
:target
element is pushed offscreen to the right, but a scrollbar is added to its parent.container
making it visible as per our definition. A similar (and more natural) example can easily be made with vertical scrolling (which could be either the.container
own scrolling, or the default scrolling of the browser window, depending on the case).Now, in our examples for visibility (which are mostly spread around the various rules that use the definition), we tend to have cases with
left: -9999px
to verify that tools handle this case (invisible content). But iirc we do not have any test with content that appear to be invisible but can be scrolled in. That is, we never exerce the "can be scrolled in" part of the definition when validating tools implementation. As always, testing definition is a bit clumsy in ACT rules since we can only test rules that rely on them and we tend to end up either copying the same-ish test in many rules, or having somewhat partial coverage for the definitions.Anyway, should we add some test cases on rules concerned about visibility to verify that tools consider the "can be scrolled in" part of the definition?