search

web-platform-tests / interop

web-platform-tests Interop project
https://wpt.fyi/interop
315 stars 28 forks source link

Accessibility (computed role + computed label) #526

Closed zcorpan closed 9 months ago

zcorpan commented 1 year ago

Description

Fix interop bugs in the browser’s accessibility tree for the parts that are now testable in wpt as a result of the interop-2023 accessibility testing investigation area, for the benefit of users who need assistive technology and for web developers wishing to design accessible user interfaces on the web.

Specification

https://w3c.github.io/aria/ https://w3c.github.io/accname/ https://w3c.github.io/html-aam/

Open Issues

No response

Tests

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility

More specifically, the “priority 1” tests in https://github.com/web-platform-tests/interop-2023-accessibility-testing/issues/3 (some tests are open PRs, some are not yet created but should be soon). Exclude “priority 2” (graphics role, SVG, DPUB).

Current Implementations

Standards Positions

No response

Browser bug reports

No response

Developer discussions

No response

Polls & Surveys

in preliminary results from State of HTML 2023, ARIA was a common response to the question "Which existing HTML features or browser APIs are you unable to use because of browser differences or lack of support?"

Also see accessibility_pain_points responses in the same survey: https://coda.io/d/State-of-HTML-Preliminary-Results_d7UVQ_fZW0V/Other-Core-Pain-Points_suDnF#Other-Core-Pain-Points_tuN-h/r4723

Inconsistent handling of elements/components between screen readers & browsers

As a FOSS developer, I can only test for browser/OS/AT combinations I personally own and that doesn't give me enough confidence

Most of the problems are UA problems - the UAs need to be quicker to fix bugs, so that developers who implement the specs can be confident they aren't doing the wrong thing

Browser support

Segmented support between browsers and OSes, have hit many cases I can support in 2 browsers but not a third. Safari and Firefox are the usual culprit.

Inconsistency between AT and browsers on how information is exposed.

Differences between browsers and tools (e.g. NVDA vs VoiceOver)

Difference between browsers, OS and screen readers

browser/AT interop

Different browsers expose different information to different assistive technology tools in different ways. Testing accessibility is a huge burden due to the discrepancies between the AT

Existing Usage

No response

Workarounds

No response

Accessibility Impact

Positive; this is explicitly designed to improve interop of some of the most important browser accessibility features.

Privacy Impact

No response

Other

No response

foolip commented 1 year ago

For survey data and web developer demand, in preliminary results from State of HTML 2023, ARIA was a common response to the freeform question "Which existing HTML features or browser APIs are you unable to use because of browser differences or lack of support?"

dtsengchromium commented 1 year ago

I think there's issues with the tests themselves and the way they fetch data from Chrome. One issue I've written up follows.

WPT calls into webdriverio https://github.com/webdriverio/webdriverio in webdriverio/packages/devtools/src/commands/getElementComputedRole.ts accessibility.snapshot

which is Puppeteer https://github.com/puppeteer/puppeteer puppeteer/packages/puppeteer-core/src/cdp/Accessibility.ts which sends the CDP command Accessibility.getFullAXTree with the interesting option defaulted to true.

In Chrome, getFullAXTree InspectorAccessibilityAgent::getFullAXTree gets called and returns the tree back over CDP.

Back in Puppeteer, puppeteer/packages/puppeteer-core/src/cdp/Accessibility.ts Accessibility.snapshot

the tree is filtered for "interesting nodes". This amounts to calling into its own wrapper AXNode ts object isInteresting method.

That method has some definition of interesting which includes only some elements e.g. controls, leaves (with text), etc. That explains a lot of failures within the dashboard pertaining to roles.

I'm therefore not sure we can use the test results here until the underlying tooling is reliable/corrected. Specifically here, the simplest thing is for WPT to pass interesting to false https://pptr.dev/api/puppeteer.accessibility.snapshot

cookiecrook commented 1 year ago

@dtsengchromium The full AX tree dump investigation is not part of this proposal, so unless I'm misunderstanding, your comments would NOT block this Interop 2024 Focus Area proposal… Is that correct? cc @aleventhal

cookiecrook commented 1 year ago

Though I could see ARIA Reflection (Gecko bug 1858211) being a problem for Gecko if there are objections to the test expectations or spec. If so, we could exclude that test from the Focus Area too.

jcsteh commented 1 year ago

[Edited by James: Oops, I accidentally corrected this one when trying to correct my own above. Sorry, this quotation is back to the original now. - jc]

@jcsteh The full AX tree dump investigation is not part of this proposal, so unless I'm misunderstanding, your comments would NOT block this Interop 2024 Focus Area proposal… Is that correct?

I don't see any comments from me on this particular issue, so I assume you're referring to https://github.com/web-platform-tests/interop-2023-accessibility-testing/issues/51#issuecomment-1613838467? So no, those comments don't block this because it's out of scope.

You're correct that ARIA element reflection is a problem for us with regard to this focus area. This should be discussed somewhere else, but as I understand it, one of the primary use cases of that entire spec is to support references across shadow DOM, yet it doesn't actually achieve that goal because of the encapsulation problem.

There are a few other tests I think we should consider excluding from this as well for various reasons, but I'll provide more details on that once I've finished my triage.

jcsteh commented 1 year ago

There seem to be some tests where we don't have a clear path forward and no engine is passing. For example, html-aam/fragile/area-role.html and el-thead/el-tfoot/el-tbody in html-aam/table-roles.html. We'll either need to exclude these or collectively determine a path forward in order to get them passing.

dtsengchromium commented 1 year ago

@cookiecrook

My comment above was referring to the way in which trees were fetched for the tests and went into detail on the manner WPT does it. Happy to have that corrected if I missed something. The takeaway is that many of the results don't mean anything at the moment.

Separate questions are whether webdriver is the right component to base these tests off of and if so, whether its worth making contributions there. For Chrome, webdriver leverages puppeteer. Who wants to make contributions to those projects to get things working? IIRC, those projects primarily concern themselves with test automation.

Also, note that puppeteer is fetching Chrome's internal Blink side a11y tree. This is probably not the right tree to test against. For example, the role literals are internal programmatic identifiers and not stable. The name computation is incomplete since we do different things for each platform a11y api (grant it, in the minority of cases).

The platform trees are the only stable representation which we really service (e.g. NSAccessibility, IAccessible2, etc). These questions and others worth a discussion at the next meeting for a11y interop.

boazsender commented 1 year ago

I agree with @cookiecrook that https://github.com/web-platform-tests/interop-2023-accessibility-testing/issues/51 is not part of this proposal as it's written. This proposal is about closing interop gaps from WPTs that don't use the axe tree.

@dtsengchromium what you're saying makes sense, we reached the same conclusion at Bocoup when we tried to use the available accessibility tree dump through cdp with the intention of testing core-aam and realized we couldn't for the reason you've mentioned here. I think we should continue this conversation on https://github.com/web-platform-tests/interop-2023-accessibility-testing/issues/51 and discuss the best path forward for exposing the platform trees for testing aam.

I also started an issue here about continuing the 2023 accessibility investigation area into 2024, which is a place where I think we could continue this work if we don't advance it further this year: https://github.com/web-platform-tests/interop/issues/459

So the tree dumping conversation would be part of the 2024 investigation area carry over, whereas this issue is a new focus area for closing interop bugs from tests based on things like getcomputedrole. I know it's a lot of interop jargon, so apologies for the category nitpicking. Where these things fall ends up being important for the wpt interop prioritization process. That's the only reason I'm pushing on this detail.

zcorpan commented 1 year ago

I think there's some misunderstanding here; as noted in https://github.com/web-platform-tests/interop-2023-accessibility-testing/issues/65#issuecomment-1757757177 , @dtsengchromium did not comment about tests using tree dump (which don't yet exist?) but about tests using get computed role (which is implemented using getFullAXTree for Chrome).

boazsender commented 1 year ago

Ah I understand now, thanks for clarifying @zcorpan.

@dtsengchromium what do you think would the best path forward for changing the underlying tooling?

cookiecrook commented 1 year ago

Yes, thanks for clarifying… FWIW, Chromium is already passing ~92% of the tests proposed for next year, but I can see the HTML-AAM: el-code test is failing as @dtsengchromium describes.

cookiecrook commented 1 year ago

@jcsteh wrote:

I don't see any comments from me on this particular issue,

Yes, sorry. That was a GitHub autocomplete that I mistakenly confirmed. Apologies as I meant to reply to @dtsengchromium

dtsengchromium commented 1 year ago

Fixing the tooling would involve upstream working in possibly each of WebDriver, Puppeteer. I don't know if this work could be rolled up into any other efforts/FAs?

jensimmons commented 1 year ago

We should figure out how to define the scope of the tests chosen for Interop 2024, and then set the scope. Not keep letting it grow over the course of the year.

jugglinmike commented 1 year ago

@dtsengchromium Thanks for that analysis! I wanted to understand the extent of the dependency on webdriverio, and I figured that I'd be able to find it in WPT since most of the "priority 1" tests which @zcorpan references have been completed. I searched WPT for references to webdriverio, but nothing came up:

$ git log -n1 --format='%h %ci %s'
b2528beea2 2023-11-01 11:05:04 -0700 [css-tables] Make `col-change-span-bg-invalidation*.html` tests fuzzy (#42897)
$ grep -ir webdriverio . | wc -l
0
$ find . -iname '*webdriverio*' | wc -l
0

I thought maybe we're talking about the Chromium project's integration with WPT, but I couldn't find any references in Chromium's source tree, either:

$ git log -n1 --format='%h %ci %s'
a87e318028 2023-11-02 00:55:19 +0000 Cleanup "inverse-fieldtrial" trybots.
$ grep -ir webdriverio . | wc -l
0
$ find . -iname '*webdriverio*' | wc -l
0

Would you mind pointing me in the right direction?

dtsengchromium commented 1 year ago

@jugglinmike I'm not too familiar with the WPT repo and how it's set up and it's been a couple of weeks. What you're looking for in WPT is the point at which it calls get computed role (I don't recall the exact method name). I traced down a couple of layers in WPT from the test html file to a common test ts file which appeared to lead to web driver.

It does also appear like maybe WPT has support for multiple test harnesses (e.g. servo, selenium, marionette).

jugglinmike commented 1 year ago

After tracing the flow of control, I've found a couple aspects that differ from what @dtsengchromium reported:

  1. The tests appear to be instructing WPT's test runner to send WebDriver commands to Chromedriver (that is, neither webdriverio nor Puppeteer appear to be involved)
  2. It seems as though the CDP command named Accessibility.getPartialAXTree is in use (not Accessibility.getFullAXTree)

These discrepencies are important because they impact how we'll fix the problem (and plan for the fix). I'll share the complete trace in the hopes that someone can verify my research or more precisely dispute it. (It's even more likely than normal that I've made a mistake because this infrastructure has been built with extension in mind.)

[^1]: The Chromium project inserts a custom definition of test_driver_internal, but I haven't analyzed that path because we're considering the test results published on wpt.fyi.

cookiecrook commented 1 year ago

Addressing the above subthread in this tracking issue.

cookiecrook commented 1 year ago

The fragile tests should be excepted from the Interop Focus Area proposal if https://github.com/web-platform-tests/wpt/issues/39617 is not resolved.

gsnedders commented 11 months ago

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility

More specifically, the “priority 1” tests in web-platform-tests/interop-accessibility#3 (some tests are open PRs, some are not yet created but should be soon). Exclude “priority 2” (graphics role, SVG, DPUB).

This leaves, what? https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility%20and%20not%20path%3A%2Fsvg-aam%20and%20not%20path%3A%2Fdpub-aam%20%20and%20not%20path%3A%2Fgraphics-aria%20and%20not%20is%3Atentative maybe?

cookiecrook commented 11 months ago

Perhaps also not path:/html-aam/fragile (I can rename that dir tentative if that's better)... I think it's unclear what the role should return for never-rendered elements like image map > area and the blocking issue is actually on the WPT test harness, not on an implementation.

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility%20and%20not%20path%3A%2Fsvg-aam%20and%20not%20path%3A%2Fdpub-aam%20%20and%20not%20path%3A%2Fgraphics-aria%20and%20not%20is%3Atentative%20and%20not%20path%3A%2Fhtml-aam%2Ffragile%20

foolip commented 11 months ago

I've taken a look at the test failures in Chrome to understand https://github.com/web-platform-tests/interop/issues/526#issuecomment-1755733993 and https://github.com/web-platform-tests/interop/issues/526#issuecomment-1799760629.

The 28 failing tests are:

/accessibility/crashtests/svg-mouse-listener.html /accname/name/comp_embedded_control.html /accname/name/comp_hidden_not_referenced.html /accname/name/comp_label.html /accname/name/comp_name_from_content.html /accname/name/comp_text_node.html /accname/name/comp_tooltip.html /html-aam/fragile/optgroup-role.html /html-aam/roles-contextual.html /html-aam/roles-dynamic-switch.tentative.window.html /html-aam/roles-generic.html /html-aam/roles.html /html-aam/roles.tentative.html /html-aam/table-roles.html /html/dom/aria-element-reflection.html /inert/inert-inlines-around-selection-range-in-contenteditable.html /svg-aam/role/roles-generic.html /wai-aria/idlharness.window.html /wai-aria/role/contextual-roles.html /wai-aria/role/fallback-roles.html /wai-aria/role/grid-roles.html /wai-aria/role/list-roles.html /wai-aria/role/listbox-roles.html /wai-aria/role/menu-roles.html /wai-aria/role/synonym-roles.html /wai-aria/role/tab-roles.html /wai-aria/role/table-roles.html /wai-aria/role/tree-roles.html

Virtually all of these use aria-utils.js, which internally depends on test_driver.get_computed_role() and test_driver.get_computed_label(). The wpt.fyi Chrome results should end up using the implementation of this in testdriver-extra.js and executorwebdriver.py, which bottoms out in calling https://w3c.github.io/webdriver/#get-computed-role or https://w3c.github.io/webdriver/#get-computed-label.

@dtsengchromium, webdriverio or puppeteer are not involved here, and it's also not related to the work to dump the full accessibility tree. I wonder if there's been a crossing of the wires between this proposal and the investigation effort? I'm not very in touch with the details, but I hope I got the above right. It would be very useful to analyze the Chrome failures and confirm if they're real implementation bugs, or if there are indeed infrastructure issues that need resolving.

cookiecrook commented 11 months ago

@foolip @dtsengchromium

there is another issue tracking the Chrome-specific stuff here: https://github.com/web-platform-tests/interop-accessibility/issues/65

cookiecrook commented 11 months ago

Accessibility WPT meeting: Group opinion is that the Chromium implementation issue is not a blocker for this Focus Area. The bug in ChromeDriver (presuming that's where the bug is) can be fixed as part of the effort to improve the Focus Area score.

cookiecrook commented 11 months ago

The SVG-specific tests were Priority 2 for the investigation and therefore excluded from this Focus Area tests, but of the 26 SVG tests finished for the investigation…

So we could leave it up to Mozilla to determine whether to include those 26.

jcsteh commented 11 months ago

I think it's reasonable to include these. I do have an open question about one of them (also failed by Chromium) in https://github.com/w3c/svg-aam/issues/32#issuecomment-1849286821.

zcorpan commented 11 months ago

@foolip is including the SVG tests ok for Chromium? (see above)

jensimmons commented 10 months ago

Tests of display:contents for consideration to include: https://github.com/web-platform-tests/wpt/pull/43740

dtsengchromium commented 10 months ago

@foolip

Thanks for the analysis! My bad on the web driver, puppeteer paths; wasn't clear where things went in WPT.

It looks like the pruning may occur in several places. Will update here with more next week.

cookiecrook commented 10 months ago

@dtsengchromium No need to post that update here because it's not relevant to the Accessibility Interop Focus Area this issue is tracking.

There's another issue tracking the ChromeDriver bug in ExecuteGetComputedLabel. Please update that one:

cookiecrook commented 9 months ago

Minor adjustment to the issue title because not all the label stuff (img alt or html:label for example) that we are testing is defined in the AccName spec.

cookiecrook commented 9 months ago

As an aside, all the display: contents; accessibility tests from #568 are also only able to confirm testing via computedrole and computedlabel, so those tests could be rolled under this one if desired, as opposed to a separate Focus Area, but it seems fine either way.

dandclark commented 9 months ago

This proposal has been accepted as part of Interop 2024! On behalf of the entire Interop team, thank you for proposing it. You’ll be able to track progress on this topic throughout the year via the Interop 2024 dashboard.

cookiecrook commented 8 months ago

FYI some subtests in the tracked Focus Area files will be removed due to this bug:

zcorpan commented 8 months ago

@cookiecrook or @nmlapre can you file a test change proposal?

nmlapre commented 8 months ago

can you file a test change proposal?

Sure thing - filed here: https://github.com/web-platform-tests/interop/issues/645

cookiecrook commented 6 months ago