search

NeoA11y / SpeakTouch

[WIP] A free software screen reader for Android.
8 stars 0 forks source link

Speak Touch does not work in flutter apps #127

Closed Irineu333 closed 6 months ago

Irineu333 commented 7 months ago

Describe the bug:

Apparently our beloved screen reader does not work with apps made in Flutter. I did an initial investigation and it seems to be related to the isImportantForAccessibility flag

To Reproduce:

Find an app made in Flutter or make a test app in Flutter. I tested with Memo.

Expected behavior

Screenshots:

Stack trace (to dev):

More infos:

App version: 1.0.0 - DEV Android version: Irrelevant Device: Irrelevant

Irineu333 commented 7 months ago

Testing on a simple button in Flutter, for each interaction, the app registers two TYPE_VIEW_HOVER_ENTER events. Being the first the node tree of the entire screen, and the second is the button with which I interacted.

Node: In view-based and compose apps, only one event is launched per interaction.

FocusInterceptor        event: TYPE_VIEW_HOVER_ENTER
FocusInterceptor        node: {
FocusInterceptor            "className": "android.widget.FrameLayout",
FocusInterceptor            "isImportantForAccessibility": true,
FocusInterceptor            "children": [
FocusInterceptor                {
FocusInterceptor                    "className": "android.view.View",
FocusInterceptor                    "isImportantForAccessibility": false,
FocusInterceptor                    "children": [
FocusInterceptor                        {
FocusInterceptor                            "className": "android.view.View",
FocusInterceptor                            "isImportantForAccessibility": false,
FocusInterceptor                            "children": [
FocusInterceptor                                {
FocusInterceptor                                    "className": "android.view.View",
FocusInterceptor                                    "isImportantForAccessibility": false,
FocusInterceptor                                    "children": [
FocusInterceptor                                        {
FocusInterceptor                                            "className": "android.view.View",
FocusInterceptor                                            "isImportantForAccessibility": false,
FocusInterceptor                                            "children": [
FocusInterceptor                                                {
FocusInterceptor                                                    "className": "android.view.View",
FocusInterceptor                                                    "isImportantForAccessibility": false,
FocusInterceptor                                                    "children": [
FocusInterceptor                                                        {
FocusInterceptor                                                            "content": "Close",
FocusInterceptor                                                            "className": "android.widget.Button",
FocusInterceptor                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                            "children": []
FocusInterceptor                                                        },
FocusInterceptor                                                        {
FocusInterceptor                                                            "content": "Detalhes",
FocusInterceptor                                                            "className": "android.view.View",
FocusInterceptor                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                            "children": []
FocusInterceptor                                                        }
FocusInterceptor                                                    ]
FocusInterceptor                                                },
FocusInterceptor                                                {
FocusInterceptor                                                    "content": "BDD - Fundamentos",
FocusInterceptor                                                    "className": "android.view.View",
FocusInterceptor                                                    "isImportantForAccessibility": false,
FocusInterceptor                                                    "children": []
FocusInterceptor                                                },
FocusInterceptor                                                {
FocusInterceptor                                                    "content": "TESTS",
FocusInterceptor                                                    "className": "android.view.View",
FocusInterceptor                                                    "isImportantForAccessibility": false,
FocusInterceptor                                                    "children": []
FocusInterceptor                                                },
FocusInterceptor                                                {
FocusInterceptor                                                    "content": "BDD",
FocusInterceptor                                                    "className": "android.view.View",
FocusInterceptor                                                    "isImportantForAccessibility": false,
FocusInterceptor                                                    "children": []
FocusInterceptor                                                },
FocusInterceptor                                                {
FocusInterceptor                                                    "className": "android.widget.ScrollView",
FocusInterceptor                                                    "isImportantForAccessibility": false,
FocusInterceptor                                                    "children": [
FocusInterceptor                                                        {
FocusInterceptor                                                            "content": "Nicolas Nascimento",
FocusInterceptor                                                            "className": "android.widget.ImageView",
FocusInterceptor                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                            "children": [
FocusInterceptor                                                                {
FocusInterceptor                                                                    "className": "android.widget.ImageView",
FocusInterceptor                                                                    "isImportantForAccessibility": false,
FocusInterceptor                                                                    "children": []
FocusInterceptor                                                                }
FocusInterceptor                                                            ]
FocusInterceptor                                                        },
FocusInterceptor                                                        {
FocusInterceptor                                                            "content": "Materiais de apoio",
FocusInterceptor                                                            "className": "android.view.View",
FocusInterceptor                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                            "children": []
FocusInterceptor                                                        },
FocusInterceptor                                                        {
FocusInterceptor                                                            "content": "Cuidado, estes materiais são referências para outros sites que não estão sob nosso controle",
FocusInterceptor                                                            "className": "android.view.View",
FocusInterceptor                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                            "children": []
FocusInterceptor                                                        },
FocusInterceptor                                                        {
FocusInterceptor                                                            "className": "android.view.View",
FocusInterceptor                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                            "children": [
FocusInterceptor                                                                {
FocusInterceptor                                                                    "className": "android.view.View",
FocusInterceptor                                                                    "isImportantForAccessibility": false,
FocusInterceptor                                                                    "children": [
FocusInterceptor                                                                        {
FocusInterceptor                                                                            "content": "📚\nBDD in Action",
FocusInterceptor                                                                            "className": "android.widget.Button",
FocusInterceptor                                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                                            "children": []
FocusInterceptor                                                                        },
FocusInterceptor                                                                        {
FocusInterceptor                                                                            "content": "📝\nBehavior Driven Development (Wikipedia)",
FocusInterceptor                                                                            "className": "android.widget.Button",
FocusInterceptor                                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                                            "children": []
FocusInterceptor                                                                        },
FocusInterceptor                                                                        {
FocusInterceptor                                                                            "content": "📝\nBehavior-Driven Development (BDD) in iOS using Swift — Part 1",
FocusInterceptor                                                                            "className": "android.widget.Button",
FocusInterceptor                                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                                            "children": []
FocusInterceptor                                                                        },
FocusInterceptor                                                                        {
FocusInterceptor                                                                            "content": "📝\nBehavior-Driven Development (BDD) in iOS using Swift — Part 2",
FocusInterceptor                                                                            "className": "android.widget.Button",
FocusInterceptor                                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                                            "children": []
FocusInterceptor                                                                        },
FocusInterceptor                                                                        {
FocusInterceptor                                                                            "content": "📝\nBehavior-Driven Development (BDD) in iOS using Swift — Part 3",
FocusInterceptor                                                                            "className": "android.widget.Button",
FocusInterceptor                                                                            "isImportantForAccessibility": false,
FocusInterceptor                                                                            "children": []
FocusInterceptor                                                                        }
FocusInterceptor                                                                    ]
FocusInterceptor                                                                }
FocusInterceptor                                                            ]
FocusInterceptor                                                        }
FocusInterceptor                                                    ]
FocusInterceptor                                                },
FocusInterceptor                                                {
FocusInterceptor                                                    "content": "ESTUDAR AGORA",
FocusInterceptor                                                    "className": "android.widget.Button",
FocusInterceptor                                                    "isImportantForAccessibility": false,
FocusInterceptor                                                    "children": []
FocusInterceptor                                                }
FocusInterceptor                                            ]
FocusInterceptor                                        }
FocusInterceptor                                    ]
FocusInterceptor                                }
FocusInterceptor                            ]
FocusInterceptor                        }
FocusInterceptor                    ]
FocusInterceptor                }
FocusInterceptor            ]
FocusInterceptor        }
FocusInterceptor        event: TYPE_VIEW_HOVER_ENTER
FocusInterceptor        node: {
                            "content": "ESTUDAR AGORA",
                            "className": "android.widget.Button",
                            "isImportantForAccessibility": false,
                            "children": []
                        }
FocusInterceptor        event: TYPE_VIEW_HOVER_EXIT
FocusInterceptor        event: TYPE_VIEW_HOVER_EXIT

Both events are ignored: the first because it has no intrinsic content and the child nodes cannot be read as they are marked as not important for accessibility; and the second, for being marked as not important for accessibility, even though it has intrinsic content.

Irineu333 commented 7 months ago

@PatrykMis Any idea what we can do? Only the root node is marked as important for accessibility.

Irineu333 commented 7 months ago

It seems that TalkBack doesn't use the method isImportantForAccessibility()... but I'm sure it validates the flag isImportantForAccessibility, perhaps in another way, and that's why it works in Flutter apps. But how?

Irineu333 commented 7 months ago

Contrary to my convictions based on absolutely nothing, even without without validating the isImportantForAccessibility() flag, the app still ignores elements marked as not important for accessibility, maintaining the expected behavior. So, as @PatrykMis suggested (in a private conversation), we can simply stop validating it.