search

flutter / flutter

Flutter makes it easy and fast to build beautiful apps for mobile and beyond
https://flutter.dev
BSD 3-Clause "New" or "Revised" License
165.69k stars 27.36k forks source link

[Web] [iOS] Pan gestures broken after initial pinch gesture #111468

Open hohlfma opened 2 years ago

hohlfma commented 2 years ago

When using a Flutter Web-App on iPad using Safari or Chrome browser, gesture detection of pan gestures gets broken after initial pinch (zoom) gesture (performed as first gesture after app start).

It seems to be important that the very first gesture gets detected as pinch gesture (two fingers). If the very first gesture is a pan gesture (one finger only) the issue does not occur afterwards. You have to reload the page and try again in this case.

Tested + confirmed on:

Steps to Reproduce

The issue can be reproduced e.g. with InteractiveViewer builder example:

  1. Build InteractiveViewer example
flutter create --sample=widgets.InteractiveViewer.builder.1 viewer_test
cd viewer_test
flutter build web --web-renderer canvaskit
  1. Launch local webserver to host the app, e.g. using python:
cd build/web
python -m http.server 80

  1. Open the web app on iPad using Safari

  2. After app has loaded, perform a pinch zoom gesture (using two fingers) as the very first interaction with the app

  3. Perform pan/move gesture (using one finger)

Expected results:

Zoom in/out and moving of the displayed grid as usual

Actual results:

After initial zooming the displayed grid, normal pan/move gestures are broken in 9/10 times. Panning with one finger seems to act like zooming, like there is still a second finger present on the screen.

Video:

https://user-images.githubusercontent.com/21358685/189883833-75aeb7b3-c640-48f2-a658-b61d632ec948.mp4

Logging of pointerCount

I added some handlers to the InteractiveViewer to output the pointerCount:

        child: InteractiveViewer(
          onInteractionStart: ((details) {
            print("onInteractionStart: pointerCount = ${details.pointerCount}");
          }),
          onInteractionUpdate: ((details) {
            print(
              "onInteractionUpdate: pointerCount = ${details.pointerCount}",
            );
          }),
          onInteractionEnd: ((details) {
            print("onInteractionEnd: pointerCount = ${details.pointerCount}");
          }),
          ...

Broken scenario (Video: 0:00 - 0:20)

Here is the output when the gesture detector gets broken after initial pinch/zoom gesture:

  1. Pinch/zoom (as very first gesture after app load) 
    [Log] onInteractionStart: pointerCount = 2 (main.dart.js, line 14198)
[Log] onInteractionUpdate: pointerCount = 2 (main.dart.js, line 14198, x46)
...
[Log] onInteractionEnd: pointerCount = 1 (main.dart.js, line 14198)

[X] This is wrong! pointerCount remains 1 after the gesture, even if there is no finger left on screen!

  1. Pan/move gesture 
    [Log] onInteractionStart: pointerCount = 2 (main.dart.js, line 14198)
[Log] onInteractionUpdate: pointerCount = 2 (main.dart.js, line 14198)
...
[Log] onInteractionEnd: pointerCount = 1 (main.dart.js, line 14198)

[X] This is wrong! pointerCount starts with 2 but there is only one finger on the screen + pointerCount still does not return to 0 after finger has been released!

Working scenario (Video: 0:20 - 0:37)

However, when the very first gesture is a pan gesture, the issue does not occur and pinch/zoom + pan/move works as expected.

  1. Pan/move gesture (as very first gesture after app load) 
    [Log] onInteractionStart: pointerCount = 1 (main.dart.js, line 14198)
[Log] onInteractionUpdate: pointerCount = 1 (main.dart.js, line 14198, x41)
...
[Log] onInteractionEnd: pointerCount = 0 (main.dart.js, line 14198)

[√] pointerCount is zero (as expected) -> No finger left on screen

  1. Pinch/zoom gesture 
    [Log] onInteractionStart: pointerCount = 2 (main.dart.js, line 14198)
[Log] onInteractionUpdate: pointerCount = 2 (main.dart.js, line 14198, x48)
...
[Log] onInteractionEnd: pointerCount = 1 (main.dart.js, line 14198)
[Log] onInteractionStart: pointerCount = 1 (main.dart.js, line 14198)
[Log] onInteractionUpdate: pointerCount = 1 (main.dart.js, line 14198)
[Log] onInteractionEnd: pointerCount = 0 (main.dart.js, line 14198)

[√] works as expected + pointerCount is zero at the end

  1. Another pan gesture 
    [Log] onInteractionStart: pointerCount = 1 (main.dart.js, line 14198)
[Log] onInteractionUpdate: pointerCount = 1 (main.dart.js, line 14198, x39)
...
[Log] onInteractionEnd: pointerCount = 0 (main.dart.js, line 14198)

[√] works as expected + pointerCount is zero at the end

Additional notes:

flutter doctor (stable) ``` flutter doctor -v [√] Flutter (Channel stable, 3.3.1, on Microsoft Windows [Version 10.0.22000.856], locale de-DE) • Flutter version 3.3.1 on channel stable at C:\build_tools\flutter-3 • Upstream repository https://github.com/flutter/flutter.git • Framework revision 4f9d92fbbd (6 days ago), 2022-09-06 17:54:53 -0700 • Engine revision 3efdf03e73 • Dart version 2.18.0 • DevTools version 2.15.0 [√] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1) • Android SDK at C:\Users\Marvin\AppData\Local\Android\Sdk • Platform android-32, build-tools 32.1.0-rc1 • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java • Java version OpenJDK Runtime Environment (build 11.0.11+9-b60-7590822) • All Android licenses accepted. [√] Chrome - develop for the web • Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe [√] Android Studio (version 2021.1) • Android Studio at C:\Program Files\Android\Android Studio • Flutter plugin can be installed from: https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 11.0.11+9-b60-7590822) [√] VS Code (version 1.70.2) • VS Code at C:\Users\Marvin\AppData\Local\Programs\Microsoft VS Code • Flutter extension version 3.46.0 [√] Connected device (2 available) • Chrome (web) • chrome • web-javascript • Google Chrome 105.0.5195.102 • Edge (web) • edge • web-javascript • Microsoft Edge 105.0.1343.27 [√] HTTP Host Availability • All required HTTP hosts are available • No issues found! ```
flutter doctor (master) ``` flutter doctor Flutter 3.4.0-19.0.pre.221 • channel master • https://github.com/flutter/flutter.git Framework • revision bbc42632a0 (2 hours ago) • 2022-09-13 02:52:39 -0400 Engine • revision 443b3646f8 Tools • Dart 2.19.0 (build 2.19.0-191.0.dev) • DevTools 2.17.0 ```
hohlfma commented 2 years ago

Digging a bit deeper into flutter\lib\src\gestures\scale.dart in function handleEvent(PointerEvent event) shows that when it gets broken it has missed a PointerUpEvent and _pointerQueue.length stucks at 1:

Working scenario (pinch after initial pan)

[Log] handleEvent: _TransformedPointerDownEvent (main.dart.js, line 24908)
[Log] _pointerCount: _pointerPanZooms: 0, _pointerQueue: 1 (main.dart.js, line 24908)

[Log] handleEvent: _TransformedPointerDownEvent (main.dart.js, line 24908)
[Log] _pointerCount: _pointerPanZooms: 0, _pointerQueue: 2 (main.dart.js, line 24908)

[Log] handleEvent: _TransformedPointerMoveEvent (main.dart.js, line 24908)
[Log] _pointerCount: _pointerPanZooms: 0, _pointerQueue: 2 (main.dart.js, line 24908)

...

[Log] handleEvent: _TransformedPointerUpEvent (main.dart.js, line 24908)
[Log] _pointerCount: _pointerPanZooms: 0, _pointerQueue: 1 (main.dart.js, line 24908)

[Log] handleEvent: _TransformedPointerUpEvent (main.dart.js, line 24908)
[Log] _pointerCount: _pointerPanZooms: 0, _pointerQueue: 0 (main.dart.js, line 24908)

--> Detected 2x PointerDownEvent + 2x PointerUpEvent

Broken scenario (pinch as initial gesture after app start)

[Log] handleEvent: _TransformedPointerDownEvent (main.dart.js, line 24908)
[Log] _pointerCount: _pointerPanZooms: 0, _pointerQueue: 1 (main.dart.js, line 24908)

[Log] handleEvent: _TransformedPointerDownEvent (main.dart.js, line 24908)
[Log] _pointerCount: _pointerPanZooms: 0, _pointerQueue: 2 (main.dart.js, line 24908)

[Log] handleEvent: _TransformedPointerMoveEvent (main.dart.js, line 24908)
[Log] _pointerCount: _pointerPanZooms: 0, _pointerQueue: 2 (main.dart.js, line 24908)

...

[Log] handleEvent: _TransformedPointerUpEvent (main.dart.js, line 24908)
[Log] _pointerCount: _pointerPanZooms: 0, _pointerQueue: 1 (main.dart.js, line 24908)

--> Detected 2x PointerDownEvent but only 1x PointerUpEvent

huycozy commented 2 years ago

Hi @hohlfma, thanks for filing the issue. This issue is reproducible on the latest stable and master channels with provided reproduction steps (including sample code). Device info: iPhone 7 iOS 15.5

Canvaskit HTML
❌ ❌
iOS (Safari) Android (Chrome)
❌ ✅

✅: No Issue ❌: Issue reproduced

Demo https://user-images.githubusercontent.com/104349824/189912066-7204a31c-d65e-4344-9e1f-11afaa2f8200.mp4
Sample code The sample code is created from `Steps to reproduction` in https://github.com/flutter/flutter/issues/111468#issue-1371268166 ```dart import 'package:flutter/material.dart'; import 'package:vector_math/vector_math_64.dart' show Quad, Vector3; void main() => runApp(const IVBuilderExampleApp()); class IVBuilderExampleApp extends StatelessWidget { const IVBuilderExampleApp({super.key}); @override Widget build(BuildContext context) { return MaterialApp( home: Scaffold( appBar: AppBar( title: const Text('IV Builder Example'), ), body: const _IVBuilderExample(), ), ); } } class _IVBuilderExample extends StatefulWidget { const _IVBuilderExample(); @override State<_IVBuilderExample> createState() => _IVBuilderExampleState(); } class _IVBuilderExampleState extends State<_IVBuilderExample> { static const double _cellWidth = 160.0; static const double _cellHeight = 80.0; // Returns the axis aligned bounding box for the given Quad, which might not be axis aligned. Rect axisAlignedBoundingBox(Quad quad) { double xMin = quad.point0.x; double xMax = quad.point0.x; double yMin = quad.point0.y; double yMax = quad.point0.y; for (final Vector3 point in [ quad.point1, quad.point2, quad.point3, ]) { if (point.x < xMin) { xMin = point.x; } else if (point.x > xMax) { xMax = point.x; } if (point.y < yMin) { yMin = point.y; } else if (point.y > yMax) { yMax = point.y; } } return Rect.fromLTRB(xMin, yMin, xMax, yMax); } @override Widget build(BuildContext context) { return Center( child: LayoutBuilder( builder: (BuildContext context, BoxConstraints constraints) { return InteractiveViewer.builder( boundaryMargin: const EdgeInsets.all(double.infinity), builder: (BuildContext context, Quad viewport) { return _TableBuilder( cellWidth: _cellWidth, cellHeight: _cellHeight, viewport: axisAlignedBoundingBox(viewport), builder: (BuildContext context, int row, int column) { return Container( height: _cellHeight, width: _cellWidth, color: row % 2 + column % 2 == 1 ? Colors.white : Colors.grey.withOpacity(0.1), child: Align( child: Text('$row x $column'), ), ); }, ); }, ); }, ), ); } } typedef _CellBuilder = Widget Function( BuildContext context, int row, int column); class _TableBuilder extends StatelessWidget { const _TableBuilder({ required this.cellWidth, required this.cellHeight, required this.viewport, required this.builder, }); final double cellWidth; final double cellHeight; final Rect viewport; final _CellBuilder builder; @override Widget build(BuildContext context) { final int firstRow = (viewport.top / cellHeight).floor(); final int lastRow = (viewport.bottom / cellHeight).ceil(); final int firstCol = (viewport.left / cellWidth).floor(); final int lastCol = (viewport.right / cellWidth).ceil(); // This will create and render exactly (lastRow - firstRow) * (lastCol - firstCol) cells return SizedBox( // Stack needs constraints, even though we then Clip.none outside of them. // InteractiveViewer.builder always sets constrained to false, giving infinite constraints to the child. // See: https://master-api.flutter.dev/flutter/widgets/InteractiveViewer/constrained.html width: 1, height: 1, child: Stack( clipBehavior: Clip.none, children: [ for (int row = firstRow; row < lastRow; row++) for (int col = firstCol; col < lastCol; col++) Positioned( left: col * cellWidth, top: row * cellHeight, child: builder(context, row, col), ), ], ), ); } } ```
flutter doctor -v ```bash [✓] Flutter (Channel stable, 3.3.1, on macOS 12.5.1 21G83 darwin-x64, locale en-VN) • Flutter version 3.3.1 on channel stable at /Users/huynq/Documents/GitHub/flutter • Upstream repository https://github.com/flutter/flutter.git • Framework revision 4f9d92fbbd (31 hours ago), 2022-09-06 17:54:53 -0700 • Engine revision 3efdf03e73 • Dart version 2.18.0 • DevTools version 2.15.0 [✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0) • Android SDK at /Users/huynq/Library/Android/sdk • Platform android-33, build-tools 31.0.0 • ANDROID_HOME = /Users/huynq/Library/Android/sdk • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 13.3) • Xcode at /Applications/Xcode.app/Contents/Developer • Build 13E113 • CocoaPods version 1.11.3 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 2021.2) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840) [✓] IntelliJ IDEA Community Edition (version 2020.3.3) • IntelliJ at /Applications/IntelliJ IDEA CE.app • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart [✓] IntelliJ IDEA Community Edition (version 2022.1.1) • IntelliJ at /Users/huynq/Library/Application Support/JetBrains/Toolbox/apps/IDEA-C/ch-0/221.5591.52/IntelliJ IDEA CE.app • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart [✓] VS Code (version 1.71.0) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.48.0 [✓] Connected device (2 available) • macOS (desktop) • macos • darwin-x64 • macOS 12.5.1 21G83 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 105.0.5195.102 [✓] HTTP Host Availability • All required HTTP hosts are available • No issues found! ``` ```bash [✓] Flutter (Channel master, 3.4.0-19.0.pre.218, on macOS 12.5.1 21G83 darwin-x64, locale en-VN) • Flutter version 3.4.0-19.0.pre.218 on channel master at /Users/huynq/Documents/GitHub/flutter_master • Upstream repository https://github.com/flutter/flutter.git • Framework revision 967b51fbcd (2 hours ago), 2022-09-12 21:32:24 -0400 • Engine revision 2689b40c50 • Dart version 2.19.0 (build 2.19.0-191.0.dev) • DevTools version 2.17.0 [✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0) • Android SDK at /Users/huynq/Library/Android/sdk • Platform android-33, build-tools 31.0.0 • ANDROID_HOME = /Users/huynq/Library/Android/sdk • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 13.3) • Xcode at /Applications/Xcode.app/Contents/Developer • Build 13E113 • CocoaPods version 1.11.3 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 2021.2) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 11.0.12+0-b1504.28-7817840) [✓] IntelliJ IDEA Community Edition (version 2020.3.3) • IntelliJ at /Applications/IntelliJ IDEA CE.app • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart [✓] IntelliJ IDEA Community Edition (version 2022.1.1) • IntelliJ at /Users/huynq/Library/Application Support/JetBrains/Toolbox/apps/IDEA-C/ch-0/221.5591.52/IntelliJ IDEA CE.app • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart [✓] VS Code (version 1.71.0) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.48.0 [✓] Connected device (5 available) • Pixel 3a (mobile) • 964AY0WL20 • android-arm64 • Android 12 (API 32) • iPhone (mobile) • d9a94afe2b649fef56ba0bfeb052f0f2a7dae95e • ios • iOS 15.5 19F77 • iPhone 13 Pro (mobile) • 67052ED0-6AEE-4A9A-B301-EAB9874D8241 • ios • com.apple.CoreSimulator.SimRuntime.iOS-15-4 (simulator) • macOS (desktop) • macos • darwin-x64 • macOS 12.5.1 21G83 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 105.0.5195.102 [✓] HTTP Host Availability • All required HTTP hosts are available • No issues found! ```
sondrelv commented 1 year ago

Any news on this issue, and/or are there any known workarounds at the moment?

lemomar commented 1 year ago

Any news on this issue, and/or are there any known workarounds at the moment?

There aren't any, as far as I know

sondrelv commented 1 year ago

I find it surprising that there isn't more activity on this issue. It's breaking my app for a large part of its user base, and I would imagine the same goes for other developers. Is there any possibility of getting this issue prioritised?

sondrelv commented 1 year ago

A workaround for this issue is tracking which pointers are active during the first pinch, and then, when the first PointerUpEvent fires, adding another PointerUpEvent for the remaining active pointer (through GestureBinding.instance.handlePointerEvent()). This is not ideal as the user is unable to keep one finger on the screen and start panning right after the initial pinch, but it is certainly better than the alternative 😊

Kadzup commented 1 year ago

A workaround for this issue is tracking which pointers are active during the first pinch, and then, when the first PointerUpEvent fires, adding another PointerUpEvent for the remaining active pointer (through GestureBinding.instance.handlePointerEvent()). This is not ideal as the user is unable to keep one finger on the screen and start panning right after the initial pinch, but it is certainly better than the alternative 😊

@sondrelv , can you share an example of your 'solution'?

lynnyuhn commented 1 year ago

I have the swipeback issue on my flutter web app as well. Additionally, I have another similar issue occurring only on my flutter web app on iPhone. I have a flutter syncfusion grid that updates a cell with on onTap(). After a random number of taps, my menubar appears disabled and application stops working...if I then do a small swipe up on screen, the menu appears enabled again and I can continue taps. If a user doesn't notice menu has changed appearance and continues to click on a difference cell than the one causing the issue, a cell somewhere else in the grid gets updated.

UPDATE: I appears that this "freezing" only happens if user taps one cell twice in a row.

sondrelv commented 1 year ago

Here's my workaround on this issue. Sorry for the delayed response, @Kadzup.

So, the way I solved this was wrapping my affected widget in a listener to keep track of and handle relevant pointer events:

@override
Widget build(BuildContext context) {
    return Listener(

        onPointerDown: (event) async {
          // Keep track of the active pointers
          _activePointers.add(event.pointer);
          // Keep track of whether user is scaling using two fingers
          _isScaling = _activePointers.length == 2;
        },

        onPointerUp: (event) async {
          // Update overview of active pointers when user removes finger 
          _activePointers.remove(event.pointer);

          // If user is scaling, an artificial gesture is not already added and we're on iOS web:
          // Add an extra PointerUpEvent to match the remaining PointerDownEvent
          if (!_initialGesturePerformed && kIsWeb) {
            var userAgent =
                dart_html.window.navigator.userAgent.toString().toLowerCase();
            if (userAgent.contains('ios') || true) {
              Future.delayed(Duration(milliseconds: 0), () async {
                if (_activePointers.isNotEmpty &&
                    _isScaling &&
                    !_initialGesturePerformed) {
                  var up = PointerUpEvent(
                      pointer: _activePointers[0],
                      position: Offset(
                          (_mapSize['width'] / 2), (_mapSize['height'] / 2)));
                  GestureBinding.instance.handlePointerEvent(up);
                  _initialGesturePerformed = true;
                  _isScaling = false;
                }
              });
            }
          }
        },
        child: ...
Kadzup commented 1 year ago

Thank you ❤️

lynnyuhn commented 1 year ago

This is definitely an iOS issue. I found a temporary fix for this problem - https://stackoverflow.com/questions/55120331/white-panel-arrives-on-double-tap-from-bottom-in-pwa-in-standalone-mode.

Here is what I did and it fixed my problem:

In the index.html. However, this may not fix the swipe-back completely. My issue was that when user double-tapped it totally messed with ALL swiping. But still give it a try - it may solve some issues. Here is snippet of code in index.html from my actual code. I added this to my index.html.