Implement accessibility support

[twitchyliquid64]

Background

Platforms typically mediate an interface between applications with a user interface, and accessibility tools (such as a screen reader), enabling assistive technologies:

• GTK/Gnome - Accessibility Toolkit (ATK)
• Windows - Active Accessibility
• MacOS - NSAccessibility protocol/messaging

Iced should provide the building blocks for making accessible applications, and integrate them automatically where possible.

Requirements

• Integrate with native platform APIs for accessibility, and provide reasonable accessibility integration for widgets built-into iced.
  • Ability to communicate the widget hierarchy
  • Ability to communicate the type, state, and 'properties' of widgets in the hierarchy
  • Ability to query for a widget at a specific location.
  • Ability to provide updates of changes to widgets, or their location in the hierarchy
• User-code should not need to use platform-specific crates.
• User widgets should be able to provide additional information to accessibility APIs.
  • Additional or better markup/attributes
  • 'Navigations' or actions
  • Indicate relationships to logically-related widgets

Ideas

• Automatic integration
  • [ ] Track widgets in a nested hierarchy
  • [ ] For each widget in the hierarchy, automatically determining basic information that can be communicated to accessibility APIs (ie: whether its a button/label/input etc, button/label text, focus state).
  • [ ] Communicate a representation of the hierarchy, basic widget information, and mutations to the platform-specific accessibility interface. This implementation could be shared with layout debugging and unit-testing facilities.
  • [ ] Implement hit-testing.
• Widget-specific extensions
  • [ ] Allow widgets to provide additional markup or attributes to an accessibility API
  • [ ] Allow widgets to indicate actions/navigation options to an accessibility API (this could be implemented at a higher level as messages?)
  • [ ] Allow widgets to indicate relationships to logically-related widgets

Imho, this means we first need to implement:

• A widget hierarchy that lives longer than layout/draw iterations
• A stable identity for a widget in the hierarchy.

Related issues

https://github.com/hecrj/iced/issues/282

[valpackett]

GTK/Gnome - Accessibility Toolkit (ATK)

Note: GTK4 is now using AT-SPI directly, because ATK is a barely maintained thing that only adds unnecessary indirection.

[savente93]

Taking inspiration from semantic HTML and its accessibility: would it be an idea to make certain marker traits such as Header, Title, Section, List. These traits could then implement necessary accessibility functionalities that (hopefully) wouldn't impact crate users at all.

Personally, I think it also has the benefit of making the code more readable, just like semantic HTML.

[kaushalyap]

@hecrj Why is not this on the roadmap?

[feylinara]

The first version of accesskit was published a few days ago, it's probably worth keeping an eye on it. Even if iced decides to go another route they're doing stuff like upstreaming stuff to winit that's necessary to elegantly implement MSAA/UIAutomation

[Alch-Emi]

I want to link in this thread, discussing accessibility in egui, and specifically Matt Campbell (@mwcampbell)'s contribution (linked) at the bottom of the thread.

The tldr is this: egui is currently trying to do integrated TTS as a stopgap for true screenreader accessibility. However, Matt has prototyped an implementation of egui that uses the accesskit beta (alpha? prototype?) to get real screenreader accessibility everywhere accesskit supports it, including a rough fork of winit to add support.

Matt also provides a brief overview of accesskit's architecture, which sounds like it will mesh really well with the elm-style architecture iced uses:

for each frame where something in the UI has changed, the application creates a TreeUpdate, which can be either a complete tree snapshot or an incremental update, and pushes it to an AccessKit platform adapter

Other highlights:

• @follower's list of accessibility resources and 101 on writing accessible software
• ndarilek does a ton of good work in the thread, but much of it is related to tts in immediate mode libs

Addendum: Huge kudos to Matt Campbell and Nolan Darilek. I feel like I didn't emphasize this before editing this post, but y'all's work is wonderful and appreciated

---

If I may switch from "providing updates mode" to "asking questions about the projects direction mode", I might want to ask (or at least open discussion for) where iced wants/plans to take screenreader accessibility in the future. At this point, I imagine there are a few options:

• Immediately adopt accesskit (most future proof, but means dropping linux/macos support for screenreader users until those platform adapters are released) related issue: https://github.com/AccessKit/accesskit/issues/56
• Follow in egui's footprints by adopting a tts implementation (medium amount of work, not true accessibility, but probably the fastest path to a cross-platform accessible iced)
• Directly interface with platform accessibility interfaces (an insane amount of work, will probably take a long time, might be lighter weight? probably shouldn't do this)
• Something else? suggestions?

[13r0ck]

Agreed, acesskit seems promising

[mwcampbell]

@Alch-Emi Thanks for looping me in here, and sorry for the late response.

I hope to start working on AccessKit in earnest again in a few weeks. I'd call it alpha right now. If all goes according to plan, the Windows implementation should be beta-quality in a few months, and then I'll start on the Mac implementation. Other platforms will probably have to wait a while. I may be able to spend some time working on integration into iced.

[savente93]

Having accessibility would be a huge competitive advantage for me in iced as not many or even any other rust frameworks (that I'm aware of) use it at the moment. Would love to help work on this is that would be helpful :) Very happy to see this discussion!

[mwcampbell]

A quick update: AccessKit is still Windows-only, and there are still serious limitations in the Windows implementation, most notably lack of support for text editing. But one major blocker has just been resolved: the newly published accesskit_winit crate makes it straightforward to use AccessKit with winit, without requiring changes to winit itself.

[ids1024]

Widget-specific extensions Allow widgets to provide additional markup or attributes to an accessibility API Allow widgets to indicate actions/navigation options to an accessibility API (this could be implemented at a higher level as messages?) Allow widgets to indicate relationships to logically-related widgets

For an initial design for accessibility support, I think these are particularly important to consider (the other parts are just "implementation details", in theory).

This probably has to be done by adding methods to the Widget trait? How would the "indicating relationships" part handled with the widget model Iced uses?

I suppose Elm must have some kind of accessibility support, so that's worth looking at.

[ids1024]

Or actually, adding method(s) to Widget is perhaps only part of this. There are two things here:

• How does the implementation of a widget (say, Button) define its default accessibility properties?
• How can accessibility properties be set by the application author, to set things that aren't automatically handled by the widgets?

[ids1024]

It looks like https://package.elm-lang.org/packages/tesk9/accessible-html/latest is worth looking at for an elm-style way of dealing with accessibility.

[mwcampbell]

Quick update on AccessKit: The Windows adapter now has very usable text editing support, and integration into egui is now a draft PR. The Linux adapter (a pure Rust implementation of AT-SPI) is progressing nicely. Now I need to focus on the Mac adapter.

[paulGeoghegan]

Is there any updates on this?

[MichaelMcDonnell]

@paulGeoghegan, it is being worked on according to this blog post from last year.

[jackpot51]

@hecrj we have carried a fork of iced with accesskit integration for a while, for use in COSMIC. I believe the changes are in a good place to be upstreamed, and if that is desired, then my team (cc @mmstick @wash2) can work to clean them up and submit a PR.

[hecrj]

@jackpot51 Sounds good to me! Accessibility is planned for the release after the next one, which should happen relatively soon.

My idea was to bring #1849 up to speed, but I imagine that is quite outdated so I definitely will appreciate any help in that regard.

[wash2]

Ya, I will work on updating that PR :+1: