Widget Testing - Can't Cancel A Stream Subscription (Package flutter_test)

MelbourneDeveloper commented 10 months ago

Is there an existing issue for this?

[X] I have searched the existing issues
[X] I have read the guide to filing a bug

Steps to reproduce

Run the first widget test in this code sample. The test will spin indefinitely.

You can download a full sample here

Notice that second test works correctly. runAsync fixes the issue. I presume that there is some kind of issue with cancelling the subscription with the fake clock.

Expected results

cancel should cancel the subscription and test should finish successfully.

Actual results

The test just spins indefinitely. This is a nightmare scenario because you get no indication of what is happening. On my machine, the test doesn't even time out. This is because technically the test completes. You can step through the code. You can step write over the line that cancels the subscription, but it makes the test hang forever.

Also notice that if you comment out the line to cancel the subscription, the test executes fine.

This is a big problem for two reasons:

1) You get no feedback about what went wrong. If you happen to have code in your app that cancels a stream subscription, your widget test will just hang and you will have no idea why.

2) Cancelling a subscription is a common occurrence in Flutter apps, and we need to verify that it works

Using runAsync is not a solution because it dramatically slows widget testing down. But, besides that, you just have to know that this is a problem before you would even know to switch to runAsync. It can take hours and hours of time to diagnose the problem. It's especially problematic if a class you are using cancels a subscription without you knowing about it.

Code sample

```dart import 'dart:async'; import 'package:flutter_test/flutter_test.dart'; void main() { testWidgets('Can Cancel A Stream Subscription - Fake Clock', (WidgetTester tester) async { await createStreamAndCancelSubscription(); }); testWidgets('Can Cancel A Stream Subscription - runAsync', (WidgetTester tester) async { tester.runAsync(createStreamAndCancelSubscription); }); } Future createStreamAndCancelSubscription() async { final controller = StreamController.broadcast(); final subscription = controller.stream.listen((event) {}); await subscription.cancel(); } ```

Screenshots or Video

https://github.com/flutter/flutter/assets/16697547/0bcd824f-c221-4441-8d35-1b6e206ebae3

Logs

christianfindlay@192-168-1-107 flutter_application_40 % flutter test 00:25 +0: Can Cancel A Stream Subscription - Fake Clock

More Information

I use streams and a stream controllers in my widget tests all the time and they work fine. They are necessary to test the code. This isn't a problem with streams in widget tests. It's only a problem with cancelling a subscription. Also, if you don't cancel subscriptions, you often get pending timers breaking the test because things haven't finished.

Lastly, the problem appears to stem from awaiting the cancellation - not necessarily cancelling itself. If you wrap the call in unawaited, you will see that the test completes fine.

Flutter Doctor output

Doctor output

```console [✓] Flutter (Channel stable, 3.16.0, on macOS 14.1.1 23B81 darwin-arm64, locale en-AU) • Flutter version 3.16.0 on channel stable at /opt/homebrew/Caskroom/flutter/3.3.10/flutter • Upstream repository https://github.com/flutter/flutter.git • Framework revision db7ef5bf9f (4 weeks ago), 2023-11-15 11:25:44 -0800 • Engine revision 74d16627b9 • Dart version 3.2.0 • DevTools version 2.28.2 [✓] Android toolchain - develop for Android devices (Android SDK version 33.0.1) • Android SDK at /Users/christianfindlay/Library/Android/sdk • Platform android-33, build-tools 33.0.1 • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 15.0.1) • Xcode at /Applications/Xcode.app/Contents/Developer • Build 15A507 • CocoaPods version 1.11.3 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [✓] Android Studio (version 2021.3) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 11.0.13+0-b1751.21-8125866) [✓] VS Code (version 1.84.2) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.78.0 [✓] Connected device (2 available) • macOS (desktop) • macos • darwin-arm64 • macOS 14.1.1 23B81 darwin-arm64 • Chrome (web) • chrome • web-javascript • Google Chrome 119.0.6045.199 [✓] Network resources • All expected network resources are available. ```

huycozy commented 10 months ago

Thanks for filing a detailed report, @MelbourneDeveloper. But I think this is similar to https://github.com/flutter/flutter/issues/5728 (and maybe somewhat to https://github.com/flutter/flutter/issues/40074 as well). Can you check it and confirm?

MelbourneDeveloper commented 10 months ago

Thanks for filing a detailed report, @MelbourneDeveloper. But I think this is similar to https://github.com/flutter/flutter/issues/5728 (and maybe somewhat to https://github.com/flutter/flutter/issues/40074 as well). Can you check it and confirm?

Without digging deep into 5728, I can say that using await does work in widget tests, but there may be some cases where it doesn't. Perhaps awaiting the cancellation of subscriptions is one of those cases.

MelbourneDeveloper commented 10 months ago

Thanks for filing a detailed report, @MelbourneDeveloper. But I think this is similar to https://github.com/flutter/flutter/issues/5728 (and maybe somewhat to https://github.com/flutter/flutter/issues/40074 as well). Can you check it and confirm?

The second issue here (40074) has the causality wrong. It's not the periodic timer causing the issue. It's the cancellation of the subscription. I use periodic timers in widget tests and it hasn't caused an issue for me.

osaxma commented 10 months ago

@MelbourneDeveloper: but there may be some cases where it doesn't.

I did try different things and I found the following two cases to block forever as well:

  // this one is similar to this issue:
  testWidgets(
    'run forever',
    (WidgetTester tester) => Future.delayed(Duration.zero),
    // or
    // (WidgetTester tester) => (() async => await Future.delayed(Duration.zero))(),
  );

  // this one is similar to 5728
  testWidgets(
    'also run forever',
    (WidgetTester tester) async => await Future.delayed(Duration.zero),
  );

MelbourneDeveloper commented 10 months ago

Thanks for filing a detailed report, @MelbourneDeveloper. But I think this is similar to #5728 (and maybe somewhat to #40074 as well). Can you check it and confirm?

@huycozy, please see how #40074 is wrong

Comment: https://github.com/flutter/flutter/issues/40074#issuecomment-1850777307

MelbourneDeveloper commented 10 months ago

Thanks for filing a detailed report, @MelbourneDeveloper. But I think this is similar to #5728 (and maybe somewhat to #40074 as well). Can you check it and confirm?

Please see this comment about #5728 https://github.com/flutter/flutter/issues/5728#issuecomment-1850802027

These two issues may be related, but there is no way to prove/disprove that

huycozy commented 10 months ago

please see how https://github.com/flutter/flutter/issues/40074 is wrong

Thanks for your update there. Maybe this issue is the root cause of #40074. Labeling this as a separate issue for other's insights. Reproduced this issue on the latest stable and master channels.

flutter doctor -v (stable and master)

```bash [✓] Flutter (Channel stable, 3.16.3, on macOS 14.1 23B74 darwin-x64, locale en-VN) • Flutter version 3.16.3 on channel stable at /Users/huynq/Documents/GitHub/flutter • Upstream repository https://github.com/flutter/flutter.git • Framework revision b0366e0a3f (23 hours ago), 2023-12-05 19:46:39 -0800 • Engine revision 54a7145303 • Dart version 3.2.3 • DevTools version 2.28.4 [✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0) • Android SDK at /Users/huynq/Library/Android/sdk • Platform android-34, build-tools 34.0.0 • ANDROID_HOME = /Users/huynq/Library/Android/sdk • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 15.0.1) • Xcode at /Applications/Xcode.app/Contents/Developer • Build 15A507 • CocoaPods version 1.13.0 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [!] Android Studio (version unknown) • Android Studio at /Applications/Android Studio Preview.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart ✗ Unable to determine Android Studio version. • Java version OpenJDK Runtime Environment (build 17.0.8+0-17.0.8b1000.22-10799086) [✓] Android Studio (version 2022.3) • Android Studio at /Applications/Android Studio Giraffe Patch 3.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231) [✓] Android Studio (version 2022.2) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • android-studio-dir = /Applications/Android Studio.app/ • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694) [✓] VS Code (version 1.84.2) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.78.0 [✓] Connected device (2 available) • macOS (desktop) • macos • darwin-x64 • macOS 14.1 23B74 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 119.0.6045.199 [✓] Network resources • All expected network resources are available. ! Doctor found issues in 1 category. ``` ```bash [!] Flutter (Channel master, 3.18.0-7.0.pre.84, on macOS 14.1 23B74 darwin-x64, locale en-VN) • Flutter version 3.18.0-7.0.pre.84 on channel master at /Users/huynq/Documents/GitHub/flutter_master ! Warning: `flutter` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path. ! Warning: `dart` on your path resolves to /Users/huynq/Documents/GitHub/flutter/bin/dart, which is not inside your current Flutter SDK checkout at /Users/huynq/Documents/GitHub/flutter_master. Consider adding /Users/huynq/Documents/GitHub/flutter_master/bin to the front of your path. • Upstream repository https://github.com/flutter/flutter.git • Framework revision 887048004b (38 minutes ago), 2023-12-11 21:29:25 -0500 • Engine revision 75cfb050cd • Dart version 3.3.0 (build 3.3.0-217.0.dev) • DevTools version 2.30.0 • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades. [✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0) • Android SDK at /Users/huynq/Library/Android/sdk • Platform android-34, build-tools 34.0.0 • ANDROID_HOME = /Users/huynq/Library/Android/sdk • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694) • All Android licenses accepted. [✓] Xcode - develop for iOS and macOS (Xcode 15.0.1) • Xcode at /Applications/Xcode.app/Contents/Developer • Build 15A507 • CocoaPods version 1.13.0 [✓] Chrome - develop for the web • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome [!] Android Studio (version unknown) • Android Studio at /Applications/Android Studio Preview.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart ✗ Unable to determine Android Studio version. • Java version OpenJDK Runtime Environment (build 17.0.8+0-17.0.8b1000.22-10799086) [✓] Android Studio (version 2022.3) • Android Studio at /Applications/Android Studio Giraffe Patch 3.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231) [✓] Android Studio (version 2022.2) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/9212-flutter • Dart plugin can be installed from: 🔨 https://plugins.jetbrains.com/plugin/6351-dart • android-studio-dir = /Applications/Android Studio.app/ • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b802.4-9586694) [✓] VS Code (version 1.84.2) • VS Code at /Applications/Visual Studio Code.app/Contents • Flutter extension version 3.78.0 [✓] Connected device (3 available) • RMX2001 (mobile) • EUYTFEUSQSRGDA6D • android-arm64 • Android 11 (API 30) • macOS (desktop) • macos • darwin-x64 • macOS 14.1 23B74 darwin-x64 • Chrome (web) • chrome • web-javascript • Google Chrome 120.0.6099.71 [✓] Network resources • All expected network resources are available. ! Doctor found issues in 2 categories. ```

a-siva commented 1 week ago

Not sure what the dart:dependency is for this issue.

flutter / flutter