Interacting with WebView on Android is flaky

[bartekpacia]

Flutter's semantics is not replicated by the Android accessibility framework during flutter drive.

In practice, the biggest pain that this problem creates is the inability to interact with WebViews on Android.

flutter run

When the app is run normally (i.e through main.dart) with flutter run, everything works fine – Android's native accessibility tree matches Flutter's semantics tree.

For example, when the currently displayed screen is the main screen:

and I use http to get the native Android views, I get the following output:

http output

$ http POST localhost:8081/getNativeWidgets className=android.widget.Button HTTP/1.1 200 OK connection: keep-alive content-length: 2593 content-type: text/plain; charset=utf-8 [ { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": null, "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": null, "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": null, "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": null, "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": "Open loading screen", "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": "Open location screen", "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": "Open notifications screen", "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": "Open overlay screen", "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": "Open permissions screen", "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": "Open scrolling screen", "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": "Open webview screen", "enabled": true, "focused": false, "resourceName": "", "text": null }, { "applicationPackage": "pl.leancode.maestro.example", "childCount": 0, "children": [], "className": "android.widget.Button", "contentDescription": null, "enabled": true, "focused": false, "resourceName": "", "text": null } ]

Also, evilmaestro hierarchy prints the same.

flutter drive

When the app is run in an integration test with flutter drive, it looks like as if Flutter's semantics tree is not sent to the Android side.

To verify this behavior, I do maestro drive -t integration_test/no_op_test.dart in maestro_test's example app directory and then I use the http tool to see how Flutter's semantics tree got translated into native Android view hierarchy.

$ http POST localhost:8081/getNativeWidgets className=android.widget.Button
HTTP/1.1 200 OK
connection: keep-alive
content-length: 2
content-type: text/plain; charset=utf-8

[]

I get nothing in response.

evilmaestro hierarchy also fails in this scenario (because it just walks the accessibility tree, which apparently is non-existent.

What's interesting is that if TalkBack is enabled, then the semantics tree is sent from Flutter to Android even during flutter drive. Demo below. I run the test with maestro drive -t integration_test/webview_test.dart.

https://user-images.githubusercontent.com/40357511/188608198-eb316143-c797-45ae-b855-adc638ab1c3b.mp4

More context

• The first investigation that was conducted in #218

• 235

• 239

[bartekpacia]

This is a Flutter bug. I created an issue on flutter/flutter about this.

We're blocked until it is fixed.

[bartekpacia]

I can't believe my eyes, but a WebView test just passed on Android.

https://user-images.githubusercontent.com/40357511/200897638-c04d1a03-3a3f-49cd-8466-c4b47fccd3b2.mov

This is pretty random though. More often fails than works. Needs more investigation.

[jagadeeshpurplle]

Hi @bartekpacia, i just want to ask that you have implemented the flutter webview using package https://pub.dev/packages/webview_flutter right? and still interacting with it doesn't work?

[bartekpacia]

@jagadeeshpurplle, yes we're using that package. You can see it here.

[akagupta9]

any updates on it?

[azeunkn0wn]

Still no progress for this issue? :(

Is there an alternative? like can you make the app blindly tap on a specific (X, Y) location of the screen and type text on it? I'm new to Integration testing, and I'm not sure if functions like that exist.

[bartekpacia]

Hi, sorry for the lack of updates.

The temporary fix is to use a different frame policy:

patrolTest(
    'test description',
    nativeAutomation: true,
    framePolicy: LiveTestWidgetsFlutterBindingFramePolicy.benchmarkLive, // <-- use this
    (PatrolTester $) async {
    // ...
  },
);

Please note that this may have some strange effects that I've not investigated yet. If you face difficulties with this workaround, please share them here and I'll try to help.

[bartekpacia]

Update

Since we migrated away from using flutter_driver a long time ago (February 2023), the bug in Flutter is no longer a blocker for us.

Here's patrol develop -t integration_test/webview_hackernews_test.dart, running on latest master:

https://github.com/leancodepl/patrol/assets/40357511/32d11953-52a2-49b1-82cf-5d9bac684c0d

But patrol test -t integration_test/webview_hackernews_test.dart does not work:

• the WebView contents aren't "pumped". I have to touch and move a finger a bit so that it's pumped.

https://github.com/leancodepl/patrol/assets/40357511/3e4b4993-5f14-49ac-a815-70b75a842816

The source of these inconsistencies in behavior between patrol test and patrol develop must be found.

[bartekpacia]

Possible temporary workarounds:

• subscribe to "WebView finished loading" callback and then progress with the test
• add a dummy 1s or 2s wait (with pumping) after webview is loaded

Post-factum edit: none of the above worked, the cause of the problem was much deeper - see #1398.

[bartekpacia]

This was fixed by #1398 on newer Android versions. It works quite reliably since Android 10 (API 29). Unfortunately, on older versions it still fails 100% of the time. This might be related to how Flutter integrates with platform views on Android.

Texture Layer Hybrid Composition has much better performance since Android 10. Under certain circumstances, fallbacks to Hybrid Composition or Virtual Display might occur. This requires further, deep investigation.

[bartekpacia]

Renamed to reflect current situation.

[bartekpacia]

Reported the bug to Flutter - see https://github.com/flutter/flutter/issues/130872.