Improve accessibility with Orca with the announcement signal

[joanmarie]

Steps to reproduce: Scenario 1

1. Launch Orca
2. Type "1+2=3"
3. Hit return

Expected results: Orca would announce the newly displayed "Malformed expression"

Actual results: Orca does not announce "Malformed expression"

The reason why is that Orca ignores most text-changed accessibility events if they come from some object other than what has focus. That's a feature; not a bug. ;) Jokes aside heuristically identifying when text changes from non-focused objects should be presented is hard to get right and hard to maintain.

A way to solve this without adding sad heuristics to Orca would be to use the new(ish) accessibility signal. See https://gitlab.gnome.org/GNOME/orca/-/blob/main/README-APPLICATION-DEVELOPERS.md#announcement-example-gtk-3

Another option might be to set the error accessible as the description-for the display accessible. See https://gitlab.gnome.org/GNOME/orca/-/blob/main/README-APPLICATION-DEVELOPERS.md#technique-use-an-accessible-description

@cwendling: Thoughts?

[cwendling]

I'll try to look into this. But just from the top of my head, can't we instead mark the relevant widget as "live" or whatever, so changes to it aren't ignored? Manually firing events for a11y doesn't struck me as the most future-proof solution when a codebase isn't audited for a11y regularly…

But maybe I can implement that myself something like (fully pseudo-code)

on_text_notify(label, pspec):
    if not label.has_focus():
        label.get_accessible.emit("notification", label.get_text(), Atk.Live.POLITE)
label.connect('notify::text', on_text_notify)

[joanmarie]

The problem with live regions in their current format is that it can be really hard to tell if text was added to an element because it is a change that should be presented to the user versus, for example, an irrelevant DOM mutation. Orca does not always get it right. Heuristics can be hard. :frowning_face:

That awfulness -- and app developers asking me "how can I make Orca speak what I want?" -- motivated the announcement/notification signal. My assumption is that the app knows when it has a message to show to the user.

All of that said, I see your point about future-proof. So I'm open to other options. BUT things to keep in mind include:

• Orca's ARIA live region support is currently web only -- and extremely web-centric. I have plans to make it global, but that's going to be a huge and ugly task.
• The solution ideally will also be something that Gtk4 developers will agree to. Ditto for Qt developers.

And that said, thank you as always for working on these things!!

[cwendling]

@joanmarie here you go: #224. Seems to work nicely, but I take any tests :)

We still have a weird behavior where the equation entry says "invalid" when getting focus, but that's unrelated. By any chance, would you know from the top of your head why though, so I don't have to search for this? :grinning:

[cwendling]

The problem with live regions in their current format is that it can be really hard to tell if text was added to an element because it is a change that should be presented to the user versus, for example, an irrelevant DOM mutation. Orca does not always get it right. Heuristics can be hard. ☹️

I understand… however, it could possibly not be inherited so hierarchy changes don't affect new children. But I get that e.g. adding elements to a list is an interesting change as well, so possibly more thought it required.

That awfulness -- and app developers asking me "how can I make Orca speak what I want?" -- motivated the announcement/notification signal. My assumption is that the app knows when it has a message to show to the user.

My fear is that app developers actually have no idea what the screen reader should speak or not. I've seen many times some people wanting to speak some things in good faith, but it actually wasn't necessarily a good idea -- not to mention what the user wants to hear or not might depend on preferences and expertise.

I'm also afraid it makes "hybrid" apps, in that they are partially read by the screen reader through their hierarchy and focus, and partially by the app's own speaking logic. And it also adds more ties between the screen reader and the app's behaviors.

All of that said, I see your point about future-proof. So I'm open to other options. BUT things to keep in mind include:

• Orca's ARIA live region support is currently web only -- and extremely web-centric. I have plans to make it global, but that's going to be a huge and ugly task.
• The solution ideally will also be something that Gtk4 developers will agree to. Ditto for Qt developers.

Yeah sure. What went through my mind was pretty basic, e.g. an additional property/attribute that the screen reader merely looks for to decide whether the event is interesting or not. But I'd have to study the direction things are going to see if more is required.

[cwendling]

OK I made a dumb mistake before so couldn't get the description trick to work, but it now does in #225. Any reason to like it less? @joanmarie

[joanmarie]

I don't have time at the moment to build and try it, but description sounds good. Did you test to verify that the description is always updated?

[joanmarie]

We still have a weird behavior where the equation entry says "invalid" when getting focus, but that's unrelated. By any chance, would you know from the top of your head why though, so I don't have to search for this? 😀

Confirmed. Will look into it.

[cwendling]

I don't have time at the moment to build and try it, but description sounds good. Did you test to verify that the description is always updated?

Yes, it's updated just as the view underneath it is, e.g. reset to empty when modifying the expression (tested with malformed expression from your example, long calculations using a ridiculous factorial (12345678!) and factorization with an expression (e.g. 123 and Ctrl+F twice)) and Orca speaks it both when it changes and when focusing the entry.

[joanmarie]

The reason Orca is saying "invalid" is because it relies on ATK to get the localized role name. Specifically, given accessible object obj:

1. Atspi.role_get_name(obj) returns the non-localized rolename nonlocalized.
2. Atk.role_for_name(nonlocalized) returns the Atk role atkrole.
3. Atk.role_get_localized_name(atkrole) returns what should be presented to the user.

This fails for status bar because it's Atspi.Role.STATUS_BAR, but Atk.Role.STATUSBAR. Orca was handling that manually. Seems the same thing (in reverse) is happening with editbar, namely it's Atspi.Role.EDITBAR but Atk.Role.EDIT_BAR. :roll_eyes:

Fixed it https://gitlab.gnome.org/GNOME/orca/-/commit/bbc36cd438ca and cherry-picked it to the gnome-46 branch.

[cwendling]

Argh. Thanks for the fix! I wonder if this couldn't be tested for (e.g. make sure in CI we don't hit invalid for valid ATSPI roles)… or if we couldn't have a non-hacky way of doing this.