search

flutter / flutter

Flutter makes it easy and fast to build beautiful apps for mobile and beyond
https://flutter.dev
BSD 3-Clause "New" or "Revised" License
163.19k stars 26.85k forks source link

WidgetTester.dragUntilVisible() to show helpful errors #89336

Open mleonhard opened 2 years ago

mleonhard commented 2 years ago

Use case

I am developing integration tests for a mobile app. The tests use WidgetTester.dragUntilVisible() (inherited from WidgetController.dragUntilVisible to scroll a ListView until a widget is visible. The method failed in my tests in several ways:

  1. When it cannot find the target widget, it logs a useless exception. It does the same thing when the widget is scrolled up out of view.

    Bad state: No element

When the exception was thrown, this was the stack:
#0      Iterable.single (dart:core/iterable.dart:498:25)
#1      WidgetController.element (package:flutter_test/src/controller.dart:112:30)
#2      WidgetController.dragUntilVisible.<anonymous closure> (package:flutter_test/src/controller.dart:1181:38)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
    integration_test/example_test.dart ```dart import 'package:flutter/cupertino.dart' show CupertinoApp; import 'package:flutter/widgets.dart' show ListView, Offset, SizedBox, Text, ValueKey, Widget; import 'package:flutter_test/flutter_test.dart' show find, testWidgets, WidgetTester; import 'package:integration_test/integration_test.dart' show IntegrationTestWidgetsFlutterBinding; void main() { IntegrationTestWidgetsFlutterBinding.ensureInitialized(); testWidgets("test1", (WidgetTester tester) async { await tester.pumpWidget( CupertinoApp( home: ListView( key: const ValueKey('table'), children: [ SizedBox(width: 10, height: 2000), Text('hello', key: ValueKey('target')), ], ), ), ); await tester.dragUntilVisible( find.byKey(ValueKey('NONEXISTENT')), find.byKey(ValueKey('table')), Offset(0.0, -50.0), ); }); } ```

  2. When the target widget is not visible and it fails to find the view widget, it throws a good exception:

    The finder "zero widgets with key [<'NONEXISTENT'>] (ignoring
offstage widgets)" (used in a call to "drag()") could not find
any matching widgets.
    integration_test/example_test.dart ```dart import 'package:flutter/cupertino.dart' show CupertinoApp; import 'package:flutter/widgets.dart' show ListView, Offset, SizedBox, Text, ValueKey, Widget; import 'package:flutter_test/flutter_test.dart' show find, testWidgets, WidgetTester; import 'package:integration_test/integration_test.dart' show IntegrationTestWidgetsFlutterBinding; void main() { IntegrationTestWidgetsFlutterBinding.ensureInitialized(); testWidgets("test1", (WidgetTester tester) async { await tester.pumpWidget( CupertinoApp( home: ListView( key: const ValueKey('table'), children: [ SizedBox(width: 10, height: 2000), Text('hello', key: ValueKey('target')) ], ), ), ); await tester.dragUntilVisible( find.byKey(ValueKey('target')), find.byKey(ValueKey('NONEXISTENT')), Offset(0.0, -50.0), ); }); } ```

  3. When the target widget is already visible, it silently ignores a bad view finder. Layout changes can make the test fail later.

    integration_test/example_test.dart ```dart import 'package:flutter/cupertino.dart' show CupertinoApp; import 'package:flutter/widgets.dart' show ListView, Offset, Text, ValueKey, Widget; import 'package:flutter_test/flutter_test.dart' show find, testWidgets, WidgetTester; import 'package:integration_test/integration_test.dart' show IntegrationTestWidgetsFlutterBinding; void main() { IntegrationTestWidgetsFlutterBinding.ensureInitialized(); testWidgets("test1", (WidgetTester tester) async { await tester.pumpWidget( CupertinoApp( home: ListView( key: const ValueKey('table'), children: [ Text('hello', key: ValueKey('target')), ], ), ), ); await tester.dragUntilVisible( find.byKey(ValueKey('target')), find.byKey(ValueKey('NONEXISTENT')), Offset(0.0, -50.0), ); }); } ```

  4. When the view finder matches two widgets, the error message is useful:

    The finder "2 widgets with key [<'table'>] (ignoring offstage
widgets): [Container-[<'table'>],
ListView-[<'table'>](scrollDirection: vertical, primary: using
primary controller, AlwaysScrollableScrollPhysics, dependencies:
[MediaQuery, PrimaryScrollController])]" (used in a call to
"drag()") ambiguously found multiple matching widgets. The
"drag()" method needs a single target.

When the exception was thrown, this was the stack:
#0      WidgetController._getElementPoint (package:flutter_test/src/controller.dart:900:7)
#1      WidgetController.getCenter (package:flutter_test/src/controller.dart:836:12)
#2      WidgetController.drag (package:flutter_test/src/controller.dart:533:7)
#3      WidgetController.dragUntilVisible.<anonymous closure> (package:flutter_test/src/controller.dart:1177:15)
#4      WidgetController.dragUntilVisible.<anonymous closure> (package:flutter_test/src/controller.dart:1175:39)
#7      TestAsyncUtils.guard (package:flutter_test/src/test_async_utils.dart:71:41)
#8      WidgetController.dragUntilVisible (package:flutter_test/src/controller.dart:1175:27)
#9      main.<anonymous closure> (file:///Users/user/example4/integration_test/example_test.dart:26:18)
<asynchronous suspension>
<asynchronous suspension>
(elided 3 frames from dart:async and package:stack_trace)
    integration_test/example_test.dart ```dart import 'package:flutter/cupertino.dart' show CupertinoApp; import 'package:flutter/widgets.dart' show Container, ListView, Offset, SizedBox, Text, ValueKey, Widget; import 'package:flutter_test/flutter_test.dart' show find, testWidgets, WidgetTester; import 'package:integration_test/integration_test.dart' show IntegrationTestWidgetsFlutterBinding; void main() { IntegrationTestWidgetsFlutterBinding.ensureInitialized(); testWidgets("test1", (WidgetTester tester) async { await tester.pumpWidget( CupertinoApp( home: Container( key: const ValueKey('table'), child: ListView( key: const ValueKey('table'), children: [ SizedBox(width: 10, height: 2000), Text('hello', key: ValueKey('target')), ], ), ), ), ); await tester.dragUntilVisible( find.byKey(ValueKey('target')), find.byKey(ValueKey('table')), Offset(0.0, -50.0), ); }); } ```

Steps to reproduce:

  1. flutter create example4

  2. Write integration_test/example_test.dart with one of the examples above.

  3. Update these files:

    pubspec.yaml ```yaml name: example4 description: A new Flutter project. publish_to: 'none' version: 1.0.0+1 environment: sdk: ">=2.12.0 <3.0.0" dependencies: flutter: sdk: flutter dev_dependencies: flutter_test: sdk: flutter integration_test: sdk: flutter flutter: ```
    test_driver/integration_test.dart ```dart import 'package:integration_test/integration_test_driver.dart' show integrationDriver; Future main() => integrationDriver(); ```

  4. Execute the test: flutter drive --driver=test_driver/integration_test.dart --target=integration_test/example_test.dart -d test-sim

flutter doctor -v ``` [✓] Flutter (Channel stable, 2.2.3, on macOS 11.5.1 20G80 darwin-x64, locale en-US) • Flutter version 2.2.3 at /Users/user/.flutter • Framework revision f4abaa0735 (9 weeks ago), 2021-07-01 12:46:11 -0700 • Engine revision 241c87ad80 • Dart version 2.13.4 [✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0) • Android SDK at /Users/user/Library/Android/sdk • Platform android-31, build-tools 31.0.0 • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS • Xcode at /Applications/Xcode.app/Contents/Developer • Xcode 12.5.1, Build version 12E507 • CocoaPods version 1.10.2 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [!] Android Studio (version 2020.3) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart ✗ Unable to find bundled Java version. • Try updating or re-installing Android Studio. [✓] Connected device (3 available) • test-sim (mobile) • E1A7325E-D299-42D4-A89C-B5DD7CF7DB06 • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-5 (simulator) • iPhone 8 (mobile) • 87606CD6-83B2-433B-9458-FD6944930D2F • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-5 (simulator) • Chrome (web) • chrome • web-javascript • Google Chrome 92.0.4515.159 ! Error: User's iPhone is not connected. Xcode will continue when User's iPhone is connected. (code -13) ! Doctor found issues in 1 category. ```

Proposal

  1. When the target widget is not found, throw an exception that includes information about the target widget. This will help users debug failing tests.
  2. Always check the view finder and throw an exception if it cannot find the view widget. This will make tests more reliable.
darshankawar commented 2 years ago

Thanks for the detailed report and code samples. Used the first example and ran it on latest master and stable and get the same exception as below:

`Bad state: No element

When the exception was thrown, this was the stack:
#0      Iterable.single (dart:core/iterable.dart:498:25)
#1      WidgetController.element (package:flutter_test/src/controller.dart:112:30)
#2      WidgetController.dragUntilVisible.<anonymous closure> (package:flutter_test/src/controller.dart:1181:38)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)`
stable, master flutter doctor -v ``` [✓] Flutter (Channel stable, 2.2.3, on Mac OS X 10.15.4 19E2269 darwin-x64, locale en-GB) • Flutter version 2.2.3 at /Users/dhs/documents/fluttersdk/flutter • Framework revision f4abaa0735 (4 days ago), 2021-07-01 12:46:11 -0700 • Engine revision 241c87ad80 • Dart version 2.13.4 [✓] Android toolchain - develop for Android devices (Android SDK version 30) • Android SDK at /Users/dhs/Library/Android/sdk • Platform android-30, build-tools 30.0.3 • ANDROID_HOME = /Users/dhs/Library/Android/sdk • Java binary at: /Users/dhs/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7486908/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS • Xcode at /Applications/Xcode.app/Contents/Developer • Xcode 12.3, Build version 12C33 • CocoaPods version 1.10.1 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 4.1) • Android Studio at /Users/dhs/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7486908/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495) [✓] VS Code (version 1.57.1) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.21.0 [✓] Connected device (4 available) • SM G975F (mobile) • RZ8M802WY0X • android-arm64 • Android 10 (API 29) • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729 • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator) • macOS (desktop) • macos • darwin-x64 • Mac OS X 10.15.4 19E2269 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 91.0.4472.114 • No issues found! [✓] Flutter (Channel master, 2.6.0-1.0.pre.160, on Mac OS X 10.15.4 19E2269 darwin-x64, locale en-GB) • Flutter version 2.6.0-1.0.pre.160 at /Users/dhs/documents/fluttersdk/flutter • Upstream repository https://github.com/flutter/flutter.git • Framework revision f77c453ab6 (2 hours ago), 2021-09-02 00:58:19 -0400 • Engine revision d877a4bbcd • Dart version 2.15.0 (build 2.15.0-68.0.dev) [✓] Android toolchain - develop for Android devices (Android SDK version 30) • Android SDK at /Users/dhs/Library/Android/sdk • Platform android-30, build-tools 30.0.3 • ANDROID_HOME = /Users/dhs/Library/Android/sdk • Java binary at: /Users/dhs/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7486908/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS • Xcode at /Applications/Xcode.app/Contents/Developer • Xcode 12.5.1, Build version 12E507 • CocoaPods version 1.10.1 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 4.1) • Android Studio at /Users/dhs/Library/Application Support/JetBrains/Toolbox/apps/AndroidStudio/ch-0/202.7486908/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495) [✓] VS Code (version 1.57.1) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.21.0 [✓] Connected device (4 available) • SM G975F (mobile) • RZ8M802WY0X • android-arm64 • Android 10 (API 29) • iPhone 12 Pro Max (mobile) • A5473606-0213-4FD8-BA16-553433949729 • ios • com.apple.CoreSimulator.SimRuntime.iOS-14-3 (simulator) • macOS (desktop) • macos • darwin-x64 • Mac OS X 10.15.4 19E2269 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 91.0.4472.114 ```

Labeling it as a proposal / feature request to show meaningful / helpful errors to properly debug the failing tests.

bislerium commented 2 years ago

Same issue. Solution yet?