search

flutter / flutter

Flutter makes it easy and fast to build beautiful apps for mobile and beyond
https://flutter.dev
BSD 3-Clause "New" or "Revised" License
165.02k stars 27.19k forks source link

When using `flutter drive`, Flutter's semantics tree doesn't produce virtual Android Views #111110

Open bartekpacia opened 2 years ago

bartekpacia commented 2 years ago

Semantic info about Flutter's widgets is propagated down to the platform so that the platform knows about Flutter widgets and can use this information e.g for accessibility purposes. But during flutter drive this process doesn't happen.

Steps to Reproduce

See that the native view hierarchy exists during flutter run

  1. Clone the https://github.com/bartekpacia/missing_a11y repository.

  2. Run the app normally:

    flutter run

  3. While the app is running, open a new terminal tab and dump native Android view hierarchy:

    adb shell uiautomator dump

  4. Copy the file containing the hierarchy dump from the Android device to your machine's $PWD:

    adb pull /sdcard/window_dump.xml .

  5. Inspect the XML file. It will contain information about the Flutter widgets. See the file here.

See that the native view hierarchy doesn't exist during flutter drive

  1. Clone the https://github.com/bartekpacia/missing_a11y repository.

  2. Run the app with flutter_driver:

    flutter drive --target integration_test/app_test.dart --driver test_driver/integration_test.dart

  3. While the app is running, open a new terminal tab and dump native Android view hierarchy:

    adb shell uiautomator dump

  4. Copy the file containing the hierarchy dump from the Android device to your machine's $PWD:

    adb pull /sdcard/window_dump.xml .

  5. Inspect the XML file. It contains very little information, e.g about Views in the status bar, but no data about Flutter widgets is in it. See the file here.

Expected results

I'd expect the view hierarchy dump to be the same no matter if the app is started with flutter run or flutter drive.

Device details

I'm pretty sure it occurs on all Android versions, but I verified the above behavior on:

Additional, random context

exaby73 commented 2 years ago

Triage report

I can reproduce this issue.

Vesions reproducible on

Info

Code sample: See repo provided here https://github.com/flutter/flutter/issues/111110#issue-1364920035

Logs ##### Dump during `flutter run` ``` ``` ##### Dump during `flutter drive` ``` ```

Flutter Doctor

flutter doctor -v (Stable) ``` [✓] Flutter (Channel stable, 3.3.1, on Microsoft Windows [Version 10.0.22000.918], locale en-US) • Flutter version 3.3.1 on channel stable at C:\Users\Nabeel\fvm\versions\stable • Upstream repository https://github.com/flutter/flutter.git • Framework revision 4f9d92fbbd (29 hours ago), 2022-09-06 17:54:53 -0700 • Engine revision 3efdf03e73 • Dart version 2.18.0 • DevTools version 2.15.0 [✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0) • Android SDK at C:\android-sdk • Platform android-33, build-tools 33.0.0 • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java • Java version OpenJDK Runtime Environment (build 11.0.12+7-b1504.28-7817840) • All Android licenses accepted. [✓] Chrome - develop for the web • CHROME_EXECUTABLE = C:\Program Files\BraveSoftware\Brave-Browser\Application\brave.exe [✓] Visual Studio - develop for Windows (Visual Studio Community 2022 17.3.2) • Visual Studio at C:\Program Files\Microsoft Visual Studio\2022\Community • Visual Studio Community 2022 version 17.3.32819.101 • Windows 10 SDK version 10.0.19041.0 [✓] Android Studio (version 2021.2) • Android Studio at C:\Program Files\Android\Android Studio • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 11.0.12+7-b1504.28-7817840) [✓] VS Code (version 1.71.0) • VS Code at C:\Users\Nabeel\AppData\Local\Programs\Microsoft VS Code • Flutter extension version 3.46.0 [✓] Connected device (4 available) • sdk gphone64 x86 64 (mobile) • emulator-5554 • android-x64 • Android 13 (API 33) (emulator) • Windows (desktop) • windows • windows-x64 • Microsoft Windows [Version 10.0.22000.918] • Chrome (web) • chrome • web-javascript • unknown • Edge (web) • edge • web-javascript • Microsoft Edge 105.0.1343.27 [✓] HTTP Host Availability • All required HTTP hosts are available • No issues found! ```
flutter doctor -v (Master) ``` [✓] Flutter (Channel master, 3.4.0-19.0.pre.137, on Microsoft Windows [Version 10.0.22000.918], locale en-US) • Flutter version 3.4.0-19.0.pre.137 on channel master at C:\Users\Nabeel\fvm\versions\master • Upstream repository https://github.com/flutter/flutter.git • Framework revision 91e9c62472 (2 hours ago), 2022-09-07 23:59:26 -0400 • Engine revision 4096e133ef • Dart version 2.19.0 (build 2.19.0-177.0.dev) • DevTools version 2.17.0 [✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0) • Android SDK at C:\android-sdk • Platform android-33, build-tools 33.0.0 • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java • Java version OpenJDK Runtime Environment (build 11.0.12+7-b1504.28-7817840) • All Android licenses accepted. [✓] Chrome - develop for the web • CHROME_EXECUTABLE = C:\Program Files\BraveSoftware\Brave-Browser\Application\brave.exe [✓] Visual Studio - develop for Windows (Visual Studio Community 2022 17.3.2) • Visual Studio at C:\Program Files\Microsoft Visual Studio\2022\Community • Visual Studio Community 2022 version 17.3.32819.101 • Windows 10 SDK version 10.0.19041.0 [✓] Android Studio (version 2021.2) • Android Studio at C:\Program Files\Android\Android Studio • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 11.0.12+7-b1504.28-7817840) [✓] VS Code (version 1.71.0) • VS Code at C:\Users\Nabeel\AppData\Local\Programs\Microsoft VS Code • Flutter extension version 3.46.0 [✓] Connected device (4 available) • sdk gphone64 x86 64 (mobile) • emulator-5554 • android-x64 • Android 13 (API 33) (emulator) • Windows (desktop) • windows • windows-x64 • Microsoft Windows [Version 10.0.22000.918] • Chrome (web) • chrome • web-javascript • unknown • Edge (web) • edge • web-javascript • Microsoft Edge 105.0.1343.27 [✓] HTTP Host Availability • All required HTTP hosts are available • No issues found! ```
bartekpacia commented 2 years ago

FWIW, it also occurs on 2.10 and 3.0 (and possibly on the previous versions as well, though I didn't check.)

christopherfujino commented 2 years ago

@jonahwilliams do you know if this workflow is expected to work?

christopherfujino commented 2 years ago

cc @eliasyishak

bartekpacia commented 2 years ago

do you know if this workflow is expected to work?

I'm not @jonahwilliams but I'll ask: why wouldn't it?

jonahwilliams commented 2 years ago

I think there are probably two parts to this:

  1. Is the framework sending semantics? If neither the platform nor user code has asked for semantics to be enabled, nothing will be processed. This requires https://api.flutter.dev/flutter/flutter_test/WidgetController/ensureSemantics.html

  2. Is the platform listening? If TalkBack or other accessibility services are not enabled, then it doesn't matter what we produce because all of this information is updated in accessibilityStateChanged listeners.

bartekpacia commented 2 years ago

Yeah but when the app is run with flutter run, the virtual view hierarchy exists, even when TalkBack is disabled. I'd expect this behavior to be the same when flutter drive-ing.

bartekpacia commented 2 years ago

Just as a side note, this is a big blocker for us in Patrol, a new Flutter testing framework. See https://github.com/leancodepl/patrol/issues/244 for more details.

bartekpacia commented 1 year ago

@jonahwilliams @christopherfujino Sorry to ping you.

I see this got P4, which means it rather won't be fixed anytime soon. I'd greatly appreciate hearing some more details about this problem, so the person who'll be working to fix this in the future, be it a Googler or a Flutter community member, will have some info to start with:

If there's nobody else to fix this, I'm gonna take up the glove.

Why am I interested in this? I'm a developer of a new testing framework for Flutter: [Patrol](https://github.com/leancodepl/patrol). One of our killer features is the ability to interact with native UI (such as permission request dialogs and webview) and our users love this. Unfortunately, this problem makes interacting with WebView impossible on Android.
jonahwilliams commented 1 year ago

Have you check if this caused by the any of the flutter_driver bindings taking over flutter run functionality?

jonahwilliams commented 1 year ago

err not flutter run functionality, but the enableFlutterDriverExtension function overrides some regular binding behavior.

bartekpacia commented 1 year ago

By "flutter_driver bindings" do you mean IntegrationTestWidgetsFlutterBinding?

If I had to bet, I'd put my cents that this problem is indeed caused by IntegrationTestWidgetsFlutterBinding, because (afaik) it's the single biggest thing that differs between flutter run and flutter drive.

bartekpacia commented 1 year ago

err not flutter run functionality, but the enableFlutterDriverExtension function overrides some regular binding behavior.

When is enableFlutterDriverExtension() called when the developer does flutter drive <...>? I think I found it – IntegrationTestWidgetsFlutterBinding.initServiceExtensions().

Anyway, thanks a lot, this is already some info to get started with.

For future reference - The `enableFlutterDriverExtension()` function is defined in `package:flutter_driver`, [link here](https://github.com/flutter/flutter/blob/master/packages/flutter_driver/lib/driver_extension.dart). - [Here's a search](https://cs.github.com/flutter/flutter?q=%22enableFlutterDriverExtension%28%29%22) for `enableFlutterDriverExtension()` in the `flutter/flutter` repo.
bartekpacia commented 1 year ago

TODO Check if the problem occurs with flutter test integration_test. I expect it to occur because the same IntegrationTestWidgetsFlutterBinding is used.

jiahaog commented 1 year ago

I wonder if https://github.com/flutter/flutter/issues/106327 is a dupe of this, or the other way round.

Were you also able to find out which binding is causing the issue? It might be easier to validate if this is caused by a particular binding by taking flutter drive out of the equation. For example, you could:

  1. Modify main.dart, right at the top of main to set up the binding of interest
  2. flutter run
  3. adb shell uiautomator dump and check the output

You can repeat (1), for enableFlutterDriverExtension (though this isn't used in integration tests), and IntegrationTestWidgetsFlutterBinding.ensureInitialized(). I would also consider going up the class hierarchy and trying LiveTestWIdgetsFlutterBinding() to try and isolate the particular binding, (and then the override) that causes the issue.

Somewhat orthogonally flutter test integration_test should be used for running integration tests instead of flutter drive unless you're targeting web.

bartekpacia commented 1 year ago

Thanks for responding and suggestions. I tried the following:

  1. WidgetsFlutterBinding.ensureInitialized() (the default): good output
  2. enableFlutterDriverExtension(): good output
  3. IntegrationTestWidgetsFlutterBinding.ensureInitialized(): bad output
  4. LiveTestWidgetsFlutterBinding.ensureInitialized(): bad output
  5. AutomatedTestWidgetsFlutterBinding: couldn't get it to work, differs too much

For (3), an assertion was triggered (assert(inTest)), so I had to manually disable it in the binding's code.

Overall, it looks as if test bindings (3, 4) were missing the feature that is present in the default bindings (1).

Somewhat orthogonally flutter test integration_test should be used for running integration tests instead of flutter drive unless you're targeting web.

We'd really like to (at least for now, we're mobile-only), but unfortunately, it's currently a no-go for our testing framework because of the following problems: