Investigation: Accessibility Tree Dump API

[mzgoddard]

I’m going to be working on an accessibility tree dump experiment in Chromium as part of this investigation area. The goal is to be able to reference test a serialization (or dump) of a page’s accessibility tree debug representation. My current plan is to follow the recommendation from this doc, and add support for this to Chromium’s wptrunner using chromedriver’s goog/cdp/execute webdriver endpoint on CDP’s Accessibility.getFullAXTree. I expect to write some experimental reference tests for accessibility using that as well.

We’d love to do this in a way that could eventually be replicable across browsers. I know there is conversation happening around a future serialization format. I’m not planning to get into that, but would appreciate any other input/feedback before I get started, especially from folks who are working on accessibility testing in WebKit and Gecko. Does anyone have any input before I get started?

[cookiecrook]

Some questions/thoughts:

• Can you reference the CDP docs or source for getFullAXTree?
• Does it start with the Web View, or document, any Node, or something else?
• Possibly implementable via WebKit's dumpAccessibilityTree in LayoutTests.

[cookiecrook]

Ideally discussion of a potential API should happen in either WebDriver (or another BTT space) or AOM.

[zcorpan]

cc @jcsteh @jgraham

[cookiecrook]

@chrishtr Will you alert David Tseng as well? I recall he was interested but I was not able to find his GitHub ID.

[chrishtr]

@dtsengchromium

[dtsengchromium]

Can you reference the CDP docs or source for getFullAXTree?

CDP's wire format draws from Chromium's accessibility tree structures and can be found here https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/public/devtools_protocol/browser_protocol.pdl;l=11?q=browser_protocol.pdl%20accessibility&ss=chromium

The wire format is just JSON served over the CDP socket to a front end e.g. dev tools, Puppeteer.

Public docs here https://chromedevtools.github.io/devtools-protocol/tot/Accessibility/

Some of the properties served up above are raw POD types like strings e.g. roles. Our source of truth format lives in https://source.chromium.org/chromium/chromium/src/+/main:ui/accessibility/ax_enums.mojom?q=ax_enums.mojom&ss=chromium

Does it start with the Web View, or document, any Node, or something else?

See the various methods and their signatures e.g. get full, partial ax tree methods which can start from document or any node respectively.

Possibly implementable via WebKit's dumpAccessibilityTree in LayoutTests.

A possible approach is to agree on the protocol format including the node properties' key/value identifier names and types.

[zcorpan]

@jcsteh said we don't have tests that dump the accessible tree, but we have tests that do assertions on the accessible tree where the input is a tree-like structure.

• search for testAccessibleTree
• an example test

Also:

Our Dev Tools Accessibility panel does have a feature to dump the tree as a JSON data structure. It probably wouldn't be too difficult to write a tree dumper which could be accessed via XPCOM and thus via WebDriver.

[jcsteh]

An interop challenge here will be standardising the exposure of things other than label and role, which are already standardised. Each engine likely has different internal names for at least some of these concepts.

[cookiecrook]

@Orkon's WebDriver BiDi issue

• https://github.com/w3c/webdriver-bidi/issues/443

[boazsender]

Quick update here that @mzgoddard and I talked, and it turns out that Accessibility.getFullAXTree doesn't expose a tree that allows us to test what we wanted to test (namely the core-aam mappings). We were looking for a good alternative test surface when we saw that @dtsengchromium mentioned in https://github.com/web-platform-tests/interop/issues/526 that getFullAXTree dumps an unstable tree which we shouldn't be writing compat tests against.

@dtsengchromium, you had a good suggestion that we discuss this at the next interop accessibility testing investigation area meeting. I see the next one is Nov 7th. Adding this to that agenda now.

[cookiecrook]

@boazsender asked me to cross-reference https://github.com/WICG/aom/issues/203 (specific proposal for https://github.com/WICG/aom/issues/197)

Specifically heading 4 in https://github.com/WICG/aom/issues/203 as use cases of what might be testable in the tree dump, whether it be from the platform tree or the engine-internal tree.

[boazsender]

Specifically section 4 of aom#203, which has some interesting properties listed out that would be in the internal chromium tree.

[boazsender]

for example

• If we have a form element like a checkbox, check if it is checked in the internal tree
• All of the states and properties

[boazsender]

@cookiecrook you also mentioned there are some tree dump tests in webkit we could reference. Can you share one here?

[cookiecrook]

And some of the WebKit LayoutTests dumpAccessibilityTree usage mentioned above. Some instances...

Usually each of those .html filenames is accompanied by a -expected.txt file that contains the raw tree dump.

For example:

• https://github.com/WebKit/WebKit/blob/main/LayoutTests/accessibility/gtk/menu-with-group-child-tree.html
• https://github.com/WebKit/WebKit/blob/main/LayoutTests/accessibility/gtk/menu-with-group-child-tree-expected.txt

[cookiecrook]

@twilco may know of a better example that shows more detail in the dump, or of other ways to test those properties that could be useful...

[cookiecrook]

@boazsender you were asking about use cases... ARIA has 50+ attrs, most of which are not currently testable with WPT automation. Only role, aria-label, and aria-labelledby are testable at the moment.

[spectranaut]

Hi all -- I couldn't bring this up in the last meeting because we weren't certain of the status of the grant, now we are certain but we still can't publicly announce it! Looks like the announcement will come ~Jan.

But for now I wanted to give Bocoup a heads up. From the direction if looks like you are taking, from the last meeting and reviewing this issue, you are working towards internal, cross-browser /internal/ accessibility tree testing. That is great and doesn't conflict with our up coming project, but you did at least mention looking into external platform API tests as an alternative to the internal tree, so I wanted to make sure we don't duplicate our efforts :)

The grant we (Igalia) got includes writing a nodejs c++ add-on for querying platform accessibility APIs, with the goal to use the add-on for easy-to-maintain, automatable, cross platform and cross API tests for all the AAMs, which we sorely need (@cookiecrook -- those manual tests in the "core-aam" folder will disappear with this work). It will be developed out in the open after we finish our design goals and the grant is announced, so you will all see it soon :)

How it relates to this issue -- this is like the "dump tree" of the "external" platform APIs. IMO there is use in being able to test /both/ trees, internal and external.

[boazsender]

That's very exciting @spectranaut, and thanks for the heads up. Will definitely coordinate before doing any work on platform accessibility tree dumping. Maybe we could add something about this to the agenda for our first accessibility testing investigation area meeting in the new year?

[jcsteh]

@spectranaut, in case it's of interest, I've been working recently to add automated platform a11y testing for ATK, IAccessible2 and UI Automation to Gecko. IA2 and UIA or already landed and running on CI. ATK is being reviewed and should be landing and running soon.

Rather than writing a C++ extension, which would have required a lot of heavy lifting, we use Python to access the platform APIs. Both Linux and Windows already have excellent abstractions in Python, plus Python can evaluate expressions received as strings, so this is much easier than writing a C++ bridge which can parse commands. (There's also a nice way to access the MacOS API from Python, but we already had a Gecko specific JS framework in place for Mac, so we didn't use this.) Since our tests run in JS, the JS tests bridge to Python using a WebSocket. The tests simply ask the bridge to run snippets of Python code and the result gets serialised to JSON and returned to the JS test as a simple value.

The main benefit of this over the existing Accessible Technology Test Adapter API is that being able to execute snippets gives us a lot more flexibility. Rather than having a restricted set of pre-defined assertions, we can run any query which can be expressed in Python. This means we can exercise more complicated APIs such as text APIs.

We could do tree dumping this way too if we wanted to. However, tree dumping doesn't allow you to exercise certain APIs very well; e.g. hit testing, text and table APIs.

If this is of interest, see Mozilla bugs 1852190 and 1862534. Here's an example IA2 test. Here's an equivalent UIA test and an equivalent ATK test (not yet landed). Here's a UIA hit test which couldn't be achieved with a tree dump. Here's an ATK test (not yet landed) which exercises part of the table interface.

[spectranaut]

Wow cool work @jcsteh, thanks for the links and I'm glad you pointed this out so early in our project! I wasn't actually aware of some of these python bindings. Definitely gives us somethings to consider.

The goal is not just to be able to dump trees, but to query them with the available accessibility APIs -- exactly like your tests. I just wanted to emphasize it replaces the need for a "external dump tree" feature mentioned in this issue.

[mzgoddard]

I've posted a CL to chromium for its wpt_internal tests related to this work. It uses the mentioned CDP getFullAXTree in the original issue description. With my limited knowledge and experience in this area, I think one of the useful things out of it is testing the derived values for aria attributes related to a given role. Like in one of the tests with a role menu element, it tests for the aria-orientation derived value of vertical when the attribute is not stated in the html. I think this could be something useful to standardize like webdriver's getComputedRole and getComputedLabel.

It also tests the API's sources for the name member, which I think is analogous to the aria label. I'm less sure if testing this is useful.

[cookiecrook]

Update: @mzgoddard's first PR is

• https://github.com/web-platform-tests/wpt/pull/43773

We should discuss this investigation at the Feb 6 Interop Accessibility meeting.

[cookiecrook]

This issue has been removed from the 2024 Investigation (#90) by consensus in today's meeting.

Leaving open in case @boazsender or @lolaodelola want to continue where Z left off. If not, this can be closed in favor of the planned approach to build a tree walker using Accessible Node parent and child references, which is integrated into the new scoring for #90.