Accessibility: Support text input widgets

[mwcampbell]

A major missing feature in Slint's accessibility support is exposure of text input widgets. I recognize that this is a major design challenge. For AccessKit (i.e. the winit backend), this requires a child AccessKit node for each span of text that has distinct formatting (at minimum, each physical line); see egui for a real-world example of how this is done. Let me know if you need help with this.

[tronical]

Right now we don't support different formatting for spans of text, so this might be easier at the moment.

All text input right now is handled by TextInput items internally, so this might not be too hard to do. Something like this in build_node_without_children (untested):

        if let Some(text_input) = item.downcast::<i_slint_core::items::TextInput>() {
            builder.set_role(Role::InlineTextBox);

            // Get cursor and anchor, text for rendering, etc.
            let visual_representation = text_input.as_pin_ref().visual_representation(None);
            let text = &visual_representation.text;
            builder.set_value(text.clone());
            builder.set_character_lengths(
                text.chars().map(|ch| ch.len_utf8() as u8).collect::<Vec<_>>(),
            );

            todo!()
        }

Or would this have to be a child of a Role::TextField?

[tronical]

We've started work in trying to use cosmic-text, which might also be a good candidate for rich text. When that comes into focus, then perhaps access kit support directly in cosmic-text might be of much value for any cosmic-text user.

[mwcampbell]

Yes, the Role::InlineTextBox node has to be a child of a Role::TextField node. The text field node must be the one that has the focus when the edit widget has the keyboard focus.

[tronical]

@mwcampbell I'm curious, what's a good way to test this so that I know it's done well enough? So far I've used the screen readers in macOS, Windows, and orca on Linux. Do they have a good mode to introspect an inline text box with text selection?

[mwcampbell]

First, AccessKit's AT-SPi implementation doesn't yet include text editing support. We'll get to it soon.

On macOS, Apple has a tool called Accessibility Inspector. I believe it's part of Xcode. And on Windows, there's Accessibility Insights.

I prefer to test with an actual screen reader, but that requires being proficient enough with a screen reader to know what should happen as you interact with the text edit control.

[hunger]

@mwcampbell: I find testing accessibility to be the biggest challenge in supporting it. I did have a blind person introduce me to how he uses a computer with a braille display and a screen reader, but that was hard to follow for me. I was totally amazed that he was able to read anything from that braille line he had! I could hardly fell the knobs at all.

I used Accerciser on Linux when getting the Qt accessibiity started: I tried to make the reports on Slint similar to those of other Linux applications. I think I could not use any of the applications with just the information in the inspection tool, but I am not sure that is because I missed something or because the state of accessibility on Linux in general.

Do you happen to know any good material I could look into to improve my understanding of the topic?

[tronical]

First, AccessKit's AT-SPi implementation doesn't yet include text editing support. We'll get to it soon.

That's good to know.

On macOS, Apple has a tool called Accessibility Inspector. I believe it's part of Xcode. And on Windows, there's Accessibility Insights.

I had tried Xcode's Accessibility Inspector, but I couldn't get it to work yet. Somehow the tree is always empty.

Thanks about the hint about Accessibility Insights, I'll give that one a try - looks very promising.

[mwcampbell]

I finally have an answer to the problem with Accessibility Inspector. It turns out that an application's windows are invisible to Accessibility Inspector unless the app is run from a proper .app bundle. I don't know why, but I confirmed experimentally that that does make all the difference.

[tronical]

Oh that’s good to know. Thanks for finding out!

[mwcampbell]

Just wondering if there's anything I can do to unblock work on this issue. Once accessibility is implemented for text editing, I think slint could easily become the most robust current Rust GUI toolkit with usable accessibility support.

[mwcampbell]

In case this issue was blocked by lack of text support in AccessKit's Unix (AT-SPI) backend, that's now fixed, thanks to @DataTriny.