How do we test definitions? (Add visibility tests with (horizontal) scroll?)

[Jym77]

A problem I've been considering while trying to improve the visibility tests in Alfa…

As per our definition of visibility, content that can be scrolled into viewport is considered visible. I guess this was mostly added for vertical scrolling, which is the most frequent case, but it also covers horizontal scrolling:

<style>
  .container {
      overflow: scroll;
  }
  .target {
      position: relative;
      left: 99999px
  }
</style>

<div class="container">
  Hello World <br />
  <div class="target">
    Lorem Ipsum
  </div>
</div>

Here, the :target element is pushed offscreen to the right, but a scrollbar is added to its parent .container making it visible as per our definition. A similar (and more natural) example can easily be made with vertical scrolling (which could be either the .container own scrolling, or the default scrolling of the browser window, depending on the case).

---

Now, in our examples for visibility (which are mostly spread around the various rules that use the definition), we tend to have cases with left: -9999px to verify that tools handle this case (invisible content). But iirc we do not have any test with content that appear to be invisible but can be scrolled in. That is, we never exerce the "can be scrolled in" part of the definition when validating tools implementation. As always, testing definition is a bit clumsy in ACT rules since we can only test rules that rely on them and we tend to end up either copying the same-ish test in many rules, or having somewhat partial coverage for the definitions.

Anyway, should we add some test cases on rules concerned about visibility to verify that tools consider the "can be scrolled in" part of the definition?

[Jym77]

This touches the more global question of "how do we test definitions?"

In that case, we have 50 rules who use "visible" and this would maybe need 3-4 cases to fully test the various corner cases (maybe more). It clearly doesn't make sense to add these to every rule.

Another way could be to spread examples across rules, adding 1 to some of the rules, another one to some other rules. However, this means that tools who do not correctly support the definition would randomly fail one rule but not another, which seems a bit arbitrarily. It also makes it harder for us to ensure that the definition is correctly covered by all the test cases, since the tests cases would be spread in the rules instead of being gathered in one place. This can create a big extra maintenance cost.

We did try to add some examples of the definition for the big definitions. These are intended mostly for tool developers. However, these are not automatically testable, and as far as I know they are not shown on the WAI website either (and not really maintained…)

---

Some ideas and questions:

• Of course, we should maintain the list of examples, and in the case of "visibility by scrolling", add some examples to that.
• Should we move these lists of examples to the WAI website too?
• We could turn these lists of examples into testable statements. The easiest way with our setup would be to write a "rule" "element is visible" (and let the existing machinery turn the examples into testable pages, the rule would map to nothing and be clearly labelled as "this is just for tools tests")), but that seems both overkill and abusing the format a bit… We could also have some machinery to turn list of examples into testables webpages independently of rules, but that is maybe more work. In any case, tools vendors could test against these directly.
• We could try and maintain, within the list of test cases, a list of rules that actually use them. E.g. write in the file, together with each test case "this case is used by rules aaaaaa, bbbbbb, and cccccc". This could help us ensure coverage of the test cases by the rule set (e.g. we aim at having each case covered by at least 3 rules). But this itself would be annoying to maintain and prone to error when we change test cases in rules…

[Jym77]

Let's try to have "rules" for testing definition. Put the rules in a separate directory (not __rules). Put the test cases in a separate JSON file. We (ACT rules) will not publish any kind of result of implementation reports but that would be a resource for implementers that we provide.

Maybe we can use the web platform tests to store the tests that make sense at that level, for example they seem to already have some accessible name tests, maybe not to the full extend but we should be able to provide more test cases (and everybody benefits…) Need to investigate more in that direction. OTOH, for definitions like "visible" , WPT might not be the best place as UA typically don't care about that directly.

[tbostic32]

I'm just now finding this conversation and the associated PR, I think having this notion of testing definitions might be very helpful for our current formulation of subjective applicability where most of the subjectivity is held within definitions.

My only counter to this is that we might run into the situation where it feels like any rule with a subjectivity feels like two rules:

• One with a variety of examples testing that implementers are correctly implementing the applicability
• A second (the actual ACT rule) that tests some parts of the applicability and then the expectation.

I'm not sure if this would cause problems or create additional overhead maintenance, but is probably worth a discussion.

[Jym77]

I think we only need to test definitions separately when they are broadly used in other rules. So for now mostly the "core concepts" of visibility, and inclusion in the accessibility tree (role and name are tested by the Web Platform Tests and we should direct implementers to these). For definitions that are mostly used in one rule, it makes sense to have all the tests as part of the rule.