App is not speech the button type in Jetpack Compose

Irineu333 commented 10 months ago

Describe the bug:

Our dear reader, unlike Talkback, is not reading buttons created with Compose as buttons.

Upon investigating, I discovered that no other reader (among those we know) besides Talkback is correctly reading them as buttons. This is somewhat worrying. Could it be that no reader other than Talkback handles Compose well?

While debugging, I found out that the class type is android.view.View. I was sure that Compose would preserve the types for better compatibility with readers, with a button being android.widget.Button, a switch being android.widget.Switch etc. But instead, it's using the most basic type, making our type-based check useless. Instead, we should focus on learning how to obtain the information of the role property of Compose's semantics.

To Reproduce:

The easiest way to test is to create a test app with a button made in Compose.

Expected behavior

The Speak Touch needs to be able to identify these buttons. It's the first step to understanding how to make the app compatible with Compose. Compose is the future, there's no escaping it, so we need to put effort into this.

Screenshots:

Talkback

https://github.com/NeoA11y/SpeakTouch/assets/45833588/76da55c9-e9e3-463a-ae26-d5f11a163098

Talkback FOSS

https://github.com/NeoA11y/SpeakTouch/assets/45833588/2ce37e03-c0ad-466f-b8e8-4c5171256a6d

Jieshou

https://github.com/NeoA11y/SpeakTouch/assets/45833588/acf6a29c-73d9-4318-934b-c8933962fb2c

Prudence

https://github.com/NeoA11y/SpeakTouch/assets/45833588/5269fb61-8d2a-41c6-9b4e-80ebd09883d6

Speak Touch

https://github.com/NeoA11y/SpeakTouch/assets/45833588/de025a63-6ad2-45c6-a926-3678b92369a7

Stack trace (to dev):

More infos:

App version: 1.0.0-DEV Android version: Android 12 Device: Moto G30

Irineu333 commented 10 months ago

Talkback FOSS is reading buttons correctly, as shown in the screenshot, I'm investigating the code but haven't found anything yet.

Irineu333 commented 10 months ago

Since I didn't succeed in analyzing the source code of talkback, I tried another approach; studying the source code of jetpack compose. If I understand how compose encodes semantic properties in AccessibilityNodeInfo, then I'll be able to use them.

Investigating in the androidx repo, I found this class, AndroidComposeViewAccessibilityDelegateCompat, which is an accessibility delegate for AndroidComposeView, which I would guess is the foundation for compose ui.

Irineu333 commented 10 months ago

Investigating AndroidComposeViewAccessibilityDelegateCompat class, I discovered that the default types (class name) when it comes to compose ui are:

android.view.View (generic)
android.widget.EditText (edit field)
android.widget.TextView (text, buttons, etc)

Irineu333 commented 10 months ago

Investigating the function populateAccessibilityNodeInfoProperties, from the class AndroidComposeViewAccessibilityDelegateCompat, it seems that some roles define a value in the roleDescription property of AccessibilityNodeInfo, like Role.Tab and Role.Switch. In my tests, Role.Button doesn't set anything in roleDescription.

For editing fields (SemanticsActions.SetText), the className is set to android.widget.EditText, whereas for text fields (SemanticsProperties.Text), it is set to android.widget.TextView.

There are some situations where the legacy type is used in the className, through the extension function Role.toLegacyClassName().

Irineu333 commented 10 months ago

Legacy types mapped by the Role.toLegacyClassName() extension function:

Role.Button -> "android.widget.Button"
Role.Checkbox -> "android.widget.CheckBox"
Role.RadioButton -> "android.widget.RadioButton"
Role.Image -> "android.widget.ImageView"
Role.DropdownList -> "android.widget.Spinner"

Irineu333 commented 10 months ago

In the tests I did, debugging in the function populateAccessibilityNodeInfoProperties, it defines className to android.widget.Button , in the following code snippet.

if (role != Role.Image ||
    semanticsNode.isUnmergedLeafNode ||
    semanticsNode.unmergedConfig.isMergingSemanticsOfDescendants
) {
    info.className = className
}

I don't know why Speak Touch isn't reading it.

Irineu333 commented 10 months ago

Finally, I understood. The type is being set for a child, and Speak Touch has some conditions to read the children.

Irineu333 commented 10 months ago

I created a class to transform the node hierarchy into json, to make it easier to view. Here is the result:

Composable with Role.Button

Box(
    Modifier
        .clickable { }
        .semantics {
            role = Role.Button
        }
) {

    Text(
        text = "With Role",
    )
}

{
    "content": "null",
    "className": "android.view.View",
    "children": [
        {
            "content": "With Role",
            "className": "android.widget.TextView",
            "isReadableAsChild": true,
            "children": []
        },
        {
            "content": "null",
            "className": "android.widget.Button",
            "isReadableAsChild": false,
            "children": []
        }
    ]
}

Composable without Role.Button

Box(
    Modifier
        .clickable { }
        .semantics { }
) {

    Text(
        text = "Without Role",
    )
}

{
    "content": "null",
    "className": "android.view.View",
    "children": [
        {
            "content": "Without Role",
            "className": "android.widget.TextView",
            "isReadableAsChild": true,
            "children": []
        }
    ]
}

Irineu333 commented 10 months ago

I really didn't expect a structure like this, it doesn't make sense to me. I'm not surprised that no other reader can handle this, besides TalkBack and Speak Touch soon. Well, it's clear that we need to tweak the NodeValidator.isReadableAsChild function and the reading of the child types. We need to be careful with this change and consider various scenarios.

NeoA11y / SpeakTouch