Which states end up in the accessibility tree?

[chrishtr]

How about:

1. rendersubtree: invisible -> Present in accessibility tree while in viewport only
2. rendersubtree: invisible skip-viewport-activation -> Not in accessibility tree unless activated for some other reason
3. rendersubtree: invisible skip-activation -> Not in accessibility tree

@rakina

This way, a UI with content offscreen will not have to pay the cost of that offscreen content. Also, content that is on-screen but hidden behind a click (e.g. <details> element, modal dialog) also won't pay the cost. This makes such content behave to accessibility tools like display:none.

Also, this will hopefully reduce the ability of developers to detect whether accessibility tools are being used.

[chrishtr]

This would make accessibility different than other UA algorithms like find-in-page.

[chrishtr]

@alice

[chrishtr]

Here is a doc summarizing current implementation choices in Chromium:

https://docs.google.com/document/d/1_xqCPxplvI-cF69cwMtEhNvd62UHrCqLvBo79xnvBBQ/edit

[chrishtr]

Currently, we represent invisible content as nodes in the accessibility tree that don't have layout - in Chromium paralance, AXNodeObject vs AXLayoutObject.

Revised proposal given that:

1. rendersubtree: invisible -> Present in accessibility tree as nodes (not layout) while offscreen. Present with layout while in viewport
2. rendersubtree: invisible skip-viewport-activation -> Not in accessibility tree at all unless activated for some other reason
3. rendersubtree: invisible skip-activation -> Not in accessibility tree at all

The reason to put AXNodeObjects in the accessibility tree for #1 is that it's a signal to assistive technology that there is content offscreen, and that a "scroll into view" API on those technologies can be used to bring it into "view" for the user.

If we want to maximize performance for item 1 above, and avoid risk of exposing assistive technology to sites, then we'll have to include AXNodeObjects for such content even if some of it might have been display:none if it were rendered. For example:

<div id=A style="rendersubtree: invisible">
  <div id=B style="display:none">
     I'm visible in the accessibility tree while offscreen!
  </div>
</div>

In this scenario, element #B will have an AXNodeObject when #A is offscreen, but when #A is scrolled on-screen, `#B will disappear from the accessibility tree.

[vmpstr]

2. rendersubtree: invisible skip-viewport-activation -> Not in accessibility tree at all unless activated for some other reason

Unfortunately I'm unfamiliar with the accessibility code, but activated for some other reason in a non-AX here would mean for example find-in-page can do it. Is there an equivalent for AX? I was under the impression that if it's not in the tree at all, then it wouldn't be findable (and if it's not findable we can't activate it, so it's a bit circular here). Or is the AX equivalent of find in page the same as the non-AX find in page? In which case, we would activate and then the activated nodes would get populated into the AX tree.

[rakina]

2. rendersubtree: invisible skip-viewport-activation -> Not in accessibility tree at all unless activated for some other reason

Unfortunately I'm unfamiliar with the accessibility code, but activated for some other reason in a non-AX here would mean for example find-in-page can do it. Is there an equivalent for AX? I was under the impression that if it's not in the tree at all, then it wouldn't be findable (and if it's not findable we can't activate it, so it's a bit circular here). Or is the AX equivalent of find in page the same as the non-AX find in page? In which case, we would activate and then the activated nodes would get populated into the AX tree.

Find-in-page will be OK since it's using the same find-in-page mechanisms and it sends back the result to the accessibility technologies (I think..), but yeah I guess the accessibility technologies can't do anything to things that are not in the AX tree? cc @alice

[alice]

Many ATs implement their own find-in-page (e.g. NVDA's "search for text" feature) which is separate from the browser's find-in-page. Of course, folks using ATs may also use the browser's find-in-page feature.

I was under the impression that if it's not in the tree at all, then it wouldn't be findable (and if it's not findable we can't activate it, so it's a bit circular here).

Yeah, my understanding (from my discussions with @chrishtr and @rakina) is that in this case it would need to be explicitly activated by the author in response to a user interaction, such a clicking a button. So it would be activatable, but the author, not the user agent, is in control of when it is activated.

However, this does get a bit weird when find-in-page can find it. I did get a bit lost in those discussions trying to understand use cases when you'd use rendersubtree: invisible skip-viewport-activation in favour of display: none - perhaps someone could spell that out for me here?

[rakina]

However, this does get a bit weird when find-in-page can find it. I did get a bit lost in those discussions trying to understand use cases when you'd use rendersubtree: invisible skip-viewport-activation in favour of display: none - perhaps someone could spell that out for me here?

One example is https://github.com/wicg/virtual-scroller, where remdersubtree: invisible skip-viewport-activation is used on everything (minus a buffer) that's not in the viewport and the author manages its own locking/unlocking when scrolling etc happens.

I'm not sure about other cases, but I've seen requests/bugs filed for making text within collapsed sections to be findable (and opens the section when navigated to), so I guess there's some interest in that too? The other big use case similar to collapsed sections is tabbed/stacked UI, where clicking a "tab" will unlock the content of that tab and hide the previous tab.

[alice]

I've seen requests/bugs filed for making text within collapsed sections to be findable

Ahh yes, I could imagine this being useful for something like the mobile view for Wikipedia, for example.

So in that case, could you also give an example for the skip-activation case?

[rakina]

So in that case, could you also give an example for the skip-activation case?

If I remember correctly the skip-activation case is more for the "the content is not fully done but we want to prepare it" case where we want to add new content/update existing content and don't want to expose them in a half-done state, adding invisibility to it and calling updateRendering() on it to prepare it to get rendered, so in this case we surely don't want to include them in the AX tree. I'm not sure if there are any other big cases aside from that, maybe @chrishtr knows more?

[chrishtr]

If I remember correctly the skip-activation case is more for the "the content is not fully done but we want to prepare it" case where we want to add new content/update existing content and don't want to expose them in a half-done state, adding invisibility to it and calling updateRendering() on it to prepare it to get rendered, so in this case we surely don't want to include them in the AX tree. I'm not sure if there are any other big cases aside from that, maybe @chrishtr knows more?

It's for DOM that is not currently part of the UI show to the user at all. One example is the pre-rendered/cached view of a non-currently visible single-page-app screen that the user might navigate to soon.

Therefore this content should definitely not be in the AT.

[chrishtr]

Many ATs implement their own find-in-page (e.g. NVDA's "search for text" feature) which is separate from the browser's find-in-page. Of course, folks using ATs may also use the browser's find-in-page feature.

Does the accessibility tech somehow make use of AXNodeObject tree entries (as opposed to AXLayoutObject) for this feature? If not, then it wouldn't be useful to represent offscreen rendersubtree: invisible skip-viewport-activation content as AXNodeObjects. Either they have full layout or not there at all.

[chrishtr]

I still think the revised resolution I proposed here is a good way forward.

Accessibility tech would still be able to find-in-page virtual scroller content by using the browser's built-in one. And the concern of detection of accessibility users would be resolved.

[alice]

Hmmm I'm still thinking about this, sorry I didn't get a chance to loop back today.

[alice]

Does the accessibility tech somehow make use of AXNodeObject tree entries (as opposed to AXLayoutObject) for this feature?

I am having trouble understanding the context for this question, so perhaps I've misunderstood.

Short answer, if I understand the question, is yes. We don't distinguish between AXNodeObjects and AXLayoutObjects when computing the platform accessibility tree; the distinction is simply an implementation detail in Blink.

[alice]

It's for DOM that is not currently part of the UI show to the user at all. One example is the pre-rendered/cached view of a non-currently visible single-page-app screen that the user might navigate to soon.

Ok, so more or less semantically equivalent to hidden, but without destroying the layout tree due to display: none - is that correct?

Therefore this content should definitely not be in the AT.

Makes sense.

[chrishtr]

Short answer, if I understand the question, is yes. We don't distinguish between AXNodeObjects and AXLayoutObjects when computing the platform accessibility tree; the distinction is simply an implementation detail in Blink.

Ok. What I was trying to get at is whether there is a notion of just sending the DOM tree to the accessibility tree as opposed to the Layout tree. The former has no rendering cost in Blink, but the latter of course does. My understanding from what Rakina said in previous docs was that somehow there is a notion of sending the DOM across for content not on screen. I think she said this is how accessibility tools know to "scroll up" or "scroll down". Is that right?

And then I asked the same question about "find in page". I think your answer was that the "find in page" feature for tools such as NVDA do in fact use the "DOM" in the accessibility tree if it's there.

Ok, so more or less semantically equivalent to hidden, but without destroying the layout tree due to display: none - is that correct?

Yes, exactly.

[rakina]

Discussed this in a meeting earlier w/ @alice and @vmpstr. We concluded to exclude skip-viewport-activation and skip-activation subtrees from the AX tree. The reasoning is in cases that use skip-viewport-activation, there's probably other ways to activate the locked subtrees already, like buttons to expand collapsed sections in Wikipedia. rendersubtree: invisible subtrees are marked as "offscreen", and for now we're ok with them not having layout information (so display:none elements are part of the ax tree too).

Might reconsider in the future, but this seems enough for now.

[fergald]

From the virtual scroller POV, we have always said that content would be finable and accessible. So leaving things out of the AX tree when they are skip-viewport-activation drops that. This seems wrong to me. I can understand that it conflicts with current implementations but from a semantics point of content is in 1 of 3 states

1 visible 2 invisible and behaving as if it is not in the document at all, so not findable, not in AX tree 3 invisible but behaving as if it is in the document, so findable, in the AX tree

These should each map to a distinct value of the rendersubtree attribute, although state-3 breaks down further in terms of whether the page author delegates responsibility for viewport-triggered activation to the browser or wants to manage that itself.

In state 3, regardless of whether it is viewport-activated or not, it seems like the AX tree should run off DOM. Am I missing something that makes this bad?

[chrishtr]

In state 3, regardless of whether it is viewport-activated or not, it seems like the AX tree should run off DOM. Am I missing something that makes this bad?

It's bad because:

1. It exposes a computational difference between AX users and non-AX users, which could be a vector for a malicious site detecting AX.
2. The current plan for skip-viewport-activation is to have it entirely controlled by the developer, and no activation is ever performed by the UA. The UA only fires events at elements that it would like to activate.

Rather than try to work around this by making a mode where AX users have more insight into offscreen content in a developer-managed virtual-scroller or other UI, we should expect AX users to go through similar UA mechanisms such as a "find-in-page" AX API, or perform gestures or clicks (for the collapsed-section use case).

OTOH render-subtree: invisible elements are managed by the UA and therefore it's acceptable to expose them to AX, because render-subtree: invisible is purely a performance optimization. The UA will in these cases also have the ability to schedule work in such a way that concern (1) above is mitigated.

[fergald]

I think we agree on the goal of making it possible for an advanced developer to manage visibility purely for performance reasons and the goal is that for all purposes, the user should be entirely unaware that this is happening. So if that's not an agreed goal (at least as a north-star) then the following won't really make sense.

This means that on-screen display, find-in-page, tab navigation, AX tree and anything else we can think of should just work like normal content. There may be differences but they should be as unobtrusive as possible.

Dropping a potential large chunk of the document out of the AX tree is a very obtrusive difference for AX users. If there are things that make it difficult that's one thing but I think we should come to an agreement on whether that would be a major miss for our goal or whether it's WAI. I think it's a major miss. Do others agree?

Assuming it's a major miss, I think we should figure out how to achieve it without compromising privacy. I don't know enough about AX clients to know whether they have a concept of focus or "viewport" but maybe for example we could run off DOM and only materialize content that is in the AX client's viewport/close to it's focus? That seems like it would be computationally undetectable.

Maybe v1 doesn't achieve it, fully-managed content is going to be rare, so we have a bit of time to try fix it in post.

[chrishtr]

I think we agree on the goal of making it possible for an advanced developer to manage visibility purely for performance reasons and the goal is that for all purposes, the user should be entirely unaware that this is happening.

I agree with this goal. However, the goal of the developer having to do no work, or the AX tool author to do no work, to work really well with such content in AX tools, is only a nice-to-have if the downsides otherwise outweigh the upsides. After all, the mode we're discussing is a developer-controlled one.

Dropping a potential large chunk of the document out of the AX tree is a very obtrusive difference for AX users. If there are things that make it difficult that's one thing but I think we should come to an agreement on whether that would be a major miss for our goal or whether it's WAI. I think it's a major miss. Do others agree?

The difference is as follows. For non-AX users, the current plan is to require the developer to do something in response to activation events for the user to ever see hidden content. In other words, the UA algorithms all have to integrate with these events.

I don't think AX tools should be an exception to the need to integrate with activation events. Therefore in v1 I think we should not expose hidden content to such tools, and expect the tools/developers/browsers to work together to integrate UA algorithms into those tools in some reasonable way that doesn't have the privacy or semantic downsides I listed earlier in this issue. In the future, I can imagine AX tools adding features to understand render-subtree semantics and integrate with them by causing activation events to be fired as appropriate.

[fergald]

I think we agree on the goal of making it possible for an advanced developer to manage visibility purely for performance reasons and the goal is that for all purposes, the user should be entirely unaware that this is happening.

I agree with this goal. However, the goal of the developer having to do no work, or the AX tool author to do no work, to work really well with such content in AX tools, is only a nice-to-have if the downsides otherwise outweigh the upsides. After all, the mode we're discussing is a developer-controlled one.

One worry I have about making developers do work is not so much about the need to do work, it's fine to make people do some work to get a benefit but there's a bit difference between do some work to set things up vs having to change code or maintain separate code paths for optimized and non-optimized content because the latter makes it really hard for people to drop in 3rd-party optimizing solutions like <virtual-scroller> or to safely manipulate their optimized content using libraries written for "normal" content.

Dropping a potential large chunk of the document out of the AX tree is a very obtrusive difference for AX users. If there are things that make it difficult that's one thing but I think we should come to an agreement on whether that would be a major miss for our goal or whether it's WAI. I think it's a major miss. Do others agree?

The difference is as follows. For non-AX users, the current plan is to require the developer to do something in response to activation events for the user to ever see hidden content. In other words, the UA algorithms all have to integrate with these events.

I don't think AX tools should be an exception to the need to integrate with activation events. Therefore in v1 I think we should not expose hidden content to such tools, and expect the tools/developers/browsers to work together to integrate UA algorithms into those tools in some reasonable way that doesn't have the privacy or semantic downsides I listed earlier in this issue. In the future, I can imagine AX tools adding features to understand render-subtree semantics and integrate with them by causing activation events to be fired as appropriate.

So if AX tools already have a concept of focus or of a "visible" range of the document that has been rendered accessibly then and if the browser already knows what that currently is and what it changes then it seems like it should be part of the activation story. I don't know if that exists or not. Or does the AX tree know what parts of the tree have been accessed by the AX tool? If there is already a pattern of incremental access by the AX tools then that can translate to activation events and can avoid the all-at-once computation.

One problem I see though is that the current version is still at the implementation level, with no semantics expressed in the document. So there's nothing to distinguish whether something is invisible because it's being actively managed for optimization vs it's invisible because it's conceptually not in the document but is waiting in the wings. You can kinda tell by whether it's got a place-holder size and or not but that's a bit fishy.

Or looking at it another way, it's OK to turn off viewport activation because JS is already fully aware of the visible viewport and it's relation to the DOM elements. JS is not at all aware of an AX tool's "rendered viewport" (for whatever that might mean, e.g. what was just read out) and we have an explicit goal of not revealing that an AX tool is in use at all. So it's kind of hard to imagine how we can not put the content into the AX tree but later put it in, what's the signal to put it in? Is it just focus navigation?

[chrishtr]

One problem I see though is that the current version is still at the implementation level, with no semantics expressed in the document. So there's nothing to distinguish whether something is invisible because it's being actively managed for optimization vs it's invisible because it's conceptually not in the document but is waiting in the wings.

Correct. Activation in these cases is controlled by the developer, and the browser has to assume that the content is not meant to be presented to the user. This is another way in which putting such invisible content into the AX tree won't work, because sometimes it really is not part of the document and it's just there because the developer is doing pre-rendering or measurement on it.

Or looking at it another way, it's OK to turn off viewport activation because JS is already fully aware of the visible viewport and it's relation to the DOM elements. JS is not at all aware of an AX tool's "rendered viewport" (for whatever that might mean, e.g. what was just read out) and we have an explicit goal of not revealing that an AX tool is in use at all. So it's kind of hard to imagine how we can not put the content into the AX tree but later put it in, what's the signal to put it in? Is it just focus navigation?

The signal is AX gestures such as scroll, focus navigation, find-in-page, or click on the visible content, just like it would be for someone not using an AX tool.

[fergald]

Correct. Activation in these cases is controlled by the developer, and the browser has to assume that the content is not meant to be presented to the user. This is another way in which putting such invisible content into the AX tree won't work, because sometimes it really is not part of the document and it's just there because the developer is doing pre-rendering or measurement on it.

I actually think it's a pretty big problem that we have no semantic signal here and the browser cannot tell conceptually in the document but optimized under the control of the developer from conceptually out of the document. If find-in-page can find the content, then AX should also see the content. In fact, I would say that

• find in page
• tab navigation
• ctrl-A ctrl-C
• AX
• a user gradually scrolling through the whole page (assuming no interesting actions are triggered by scroll)

should all see experience exactly the same content. If we have a mode where one or more of those see different content, then that seems broken to me. Maybe it can ship as a v1 is a different question but the intention to make it so seems like it should be there.

The signal is AX gestures such as scroll, focus navigation, find-in-page, or click on the visible content, just like it would be for someone not using an AX tool.

The question is, are the AX tools that present content without scrolling to it?

E.g. on Android if I turn on talkback, and start swiping right (to read out more and more of a page), the scroll position also changes so that what is being read is on-screen (at least at the block level, sometimes it's reading stuff off-screen but I assume it's all part of the same div).

So talkback on Android is safe but are there other screen readers that will e.g. give you a list of and in the page and read them out to you without moving the the viewport or the focus? If so, they will be broken by this kind of content.

[alice]

If we have a mode where one or more of those see different content, then that seems broken to me.

My understanding is that skip-viewport-activation is exactly that: a user scrolling through the page would not see that content (because the viewport moving would not activate that subtree).

The example that @vmpstr gave me which helped form my intuition and my support for this initial design is the Wikipedia mobile view, where parts of the page are hidden from view in collapsed sections. Scrolling those sections into view doesn't cause them to be activated: the user must take an explicit action to activate them, unless they are activated by a UA interaction such as find in page.

If we exposed those collapsed sections of the page to assistive technologies, it would actually be creating a significantly different experience for AT users than for non-AT users. In that case, AT users would notice no difference between collapsed sections and non-collapsed sections. Quite apart from the privacy implications, that also seems like a poor experience to me.

are there other screen readers that will e.g. give you a list of and in the page and read them out to you without moving the the viewport or the focus? If so, they will be broken by this kind of content.

It seems like this is more describing the case without skip-viewport-activation or skip-activation, in which case we will be exposing content to ATs.

[fergald]

@alice There's some context that you're missing. The WP collapsed-section is definitely as you described but there is another very different use case for skip-viewport which is that of the virtual scroller. We do not believe that we can build good enough "virtualization" into the rendering engine, partly because if it's in the rendering engine, it needs to work perfectly for all cases. Instead we want people to be able to build virtual-scrollers in JS that work really well. When they do that, they can set limitations (e.g. that all content must have certain contain style or that it won't support flex layout etc) but the idea is that you can build a virtual scroller that keeps all the content in the DOM (not in JS with something creating and deleting DOM elements as the user scrolls) and just use invisibility to control the rendering cost of all this content. A lot of this is here - https://github.com/WICG/virtual-scroller.

For this kind of content, it should behave exactly as if it is in the doc, it should be find-able, tab-navigable, selectable etc and it should also be just as accessible as if it was normal content.

The non-skip viewport invisibility gets you a kind of half-way version of this. The content doesn't cost rendering until it scrolls into view but once it scrolls into view it becomes just like any other content. A virtual-scroller would re-hide content that is gone off-screen so that it won't slow down rendering but again, it's not hidden like a collapsed WP section. To the user it should be indistinguishable from unhidden content.

[alice]

@fergal Could you illustrate how you would build a virtual scroller using skip-viewport-activation? How would content become activated in this context?

[fergald]

The scroller uses IntersectionObserver to track viewport moves itself. We're currently shipping an experimental one in Chrome

https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/script/resources/layered_api/elements/virtual-scroller/

but I'm no longer working on this so it's been a stagnant and e.g. I think has not been updated to use skip-viewport-activation with the result that it can get into fights with the browser about whether elements should be visible or not.

[alice]

Can you expand on why you'd use IntersectionObserver in preference to viewport activation?

[fergald]

Because viewport activation will never re-hide something, so performance degrades as more and more content becomes visible. Current JS<->DOM virtual scrollers keep basically a screen-full of content visible so rendering/style is always O(screen size), not O(page size) but find in page, accessibility etc is destroyed. The goal is to allow people to write a virtual scroller that keeps everything in DOM, preserving FIP, accessibility but still keep rendering O(screen size) and to do that you need to re-hide content that has been hidden.

Once you need to re-hide, you need to be in the business of tracking scroll position either via IntersectionObservers or other ways (using IntersectionObservers is the result of much discussion and is not really important except for the fact that we are somehow tracking this) and once your JS has an opinion on what should be visible/invisible, you don't really want the browser butting in with it's opinion too, so you just turn off the the browser's one entirely to keep things simpler.

I'm available to VC most of this afternoon BTW if you want to get into more details.

[fergald]

Also, https://github.com/fergald/virtual-scroller-demos has instructions on how to launch chrome with the scroller enabled, you can take the demos for a spin, it appears that they still work, although there might be some issues.

[alice]

@fergald Thanks for the extra detail there.

I'm available to VC most of this afternoon BTW if you want to get into more details.

I'll get in touch on another channel.

[chrishtr]

Because viewport activation will never re-hide something

We actually changed the behavior of viewport activation to automatically re-hide. See this issue comment resolution.

Your other points about IntersectionObservers for all of the locked elements facing scaling pressure for very large content still stand though.

[alice]

Recapping an earlier in-person conversation with @fergald, as well as replying to some points raised here...

are there other screen readers that will e.g. give you a list of [headings] and in the page and read them out to you without moving the the viewport or the focus?

Yes, VoiceOver (built in screen reader on Mac devices) does this. I made a little demo with a Wikipedia page:

In the demo, I use ctrl-option-U to bring up the VoiceOver rotor, then the down arrow to scroll through the list of headings on the page. As I arrow through the headings, VoiceOver highlights them and speaks them (except when I start arrowing too fast for it to keep up; then it makes a gentle clicking noise) but does not scroll through the page.

Once I reach a heading that I'm (ostensibly) interested in, I press Enter to select it. At that point, VO scrolls to that heading.

If so, they will be broken by this kind of content.

Yes, that behaviour will not work in the virtual scroller case where skip-viewport-activation is used.

However, it's worth noting that in the case of a page too big for viewport activation to work out of the box (I understand from @fergald that a virtual scroller for smaller pages works fine with viewport activation on its own), AT will already struggle if not outright choke on the page.

For example, trying to load the WHATWG single page HTML spec in Chrome with VoiceOver turned on results in a sad tab after about a minute of "Chrome busy".

We have several existing bugs on Chrome specifically about hanging behaviour with large numbers of direct child nodes of a single node; the latest I'm aware of is https://crbug.com/981322 (public issue).

Even if browsers can handle that large a number of (child) nodes, it's still not clear that all assistive technologies are equipped to handle such a massive tree effectively.

For that reason, I'm still in favour of excluding skip-viewport-activation nodes from the accessibility tree for the time being. It's not ideal, I agree, but including them doesn't necessarily buy us very much either.

Therefore in v1 I think we should not expose hidden content to such tools, and expect the tools/developers/browsers to work together to integrate UA algorithms into those tools in some reasonable way that doesn't have the privacy or semantic downsides I listed earlier in this issue. In the future, I can imagine AX tools adding features to understand render-subtree semantics and integrate with them by causing activation events to be fired as appropriate. (@chrishtr from https://github.com/WICG/display-locking/issues/102#issuecomment-575395941)

One project we should keep an eye on is Accessible Loading and Searching of Content (yes, really).

This has the explicit goal of solving a closely related problem: pages which don't send all of their content to the client at any given time (Google Docs/MS Office 360 being typical examples).

The current, deliberately very basic iteration is to add an ARIA attribute which exposes the fact that not all of the content exists in the DOM, and hints at a direction to scroll to load more content. This allows the AT to scroll to the end of a container to prompt more loading of content.

From this base, I believe the authors of that project are hoping to build out richer two-way communication APIs between ATs and browsers around partial/incomplete content.