search

flutter / samples

A collection of Flutter examples and demos
https://flutter.github.io/samples/
Other
17.11k stars 7.5k forks source link

`testing_app` hangs on Windows #1000

Open johnpryan opened 2 years ago

johnpryan commented 2 years ago

After enabling Flutter desktop on Windows, running flutter test integration_test -d windows hangs until ctrl+c, then a TimeoutException is printed.

@dnfield @csells I followed the setup instructions at https://docs.flutter.dev/desktop and I can successfully run the app under test (flutter run -d windows)

The integration tests in this sample are currently failing on a macOS device: (https://github.com/flutter/samples/issues/999)

Logs ``` C:\Users\johnp\code\samples\testing_app>flutter test integration_test -d windows 00:00 +0: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart B21:49 +0 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart [E] TimeoutException after 0:12:00.000000: Test timed out after 12 minutes. package:test_api/src/backend/invoker.dart 333:28 Invoker._handleError. dart:async/zone.dart 1418:47 _rootRun dart:async/zone.dart 1328:19 _CustomZone.run package:test_api/src/backend/invoker.dart 331:10 Invoker._handleError package:test_api/src/backend/invoker.dart 287:9 Invoker.heartbeat.. dart:async/zone.dart 1426:13 _rootRun dart:async/zone.dart 1328:19 _CustomZone.run package:test_api/src/backend/invoker.dart 286:38 Invoker.heartbeat. dart:async-patch/timer_patch.dart 18:15 Timer._createTimer. dart:isolate-patch/timer_impl.dart 395:19 _Timer._runTimers dart:isolate-patch/timer_impl.dart 426:5 _Timer._handleMessage dart:isolate-patch/isolate_patch.dart 192:12 _RawReceivePortImpl._handleMessage 21:49 +0 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart 21:50 +0 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart Terminate batch job (Y/N)? y ``` 
C:\Users\johnp\code\samples\testing_app>flutter --version
Flutter 2.10.0-0.1.pre • channel beta • https://github.com/flutter/flutter.git
Framework • revision 628f0e3f3a (9 days ago) • 2022-01-11 19:50:05 -0600
Engine • revision 63ca99584a
Tools • Dart 2.16.0 (build 2.16.0-134.1.beta) • DevTools 2.9.2
csells commented 2 years ago

@dnfield can you take a look at this? I'd love integration_test to work well on Windows.

johnpryan commented 2 years ago

After running the tests again, I can see one of the tests passing but another gets a TestDeviceException:

logs ``` C:\Users\johnp\code\samples\testing_app>flutter test integration_test -d windows 00:00 +0: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart B00:16 +0: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart 00:26 +4: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart B00:40 +4: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart Error waiting for a debug connection: The log reader stopped unexpectedly, or never started. 00:40 +4 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart [E] TestDeviceException(Unable to start the app on the device.) package:flutter_tools/src/test/integration_test_device.dart 61:7 IntegrationTestTestDevice.start 00:40 +4 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\state_mgmt_test.dart B00:40 +4 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart [E] Bad state: Cannot add new events after calling close dart:async/broadcast_stream_controller.dart 243:24 _BroadcastStreamController.add dart:async/zone.dart 1434:47 _rootRunUnary dart:async/zone.dart 1335:19 _CustomZone.runUnary dart:async/zone.dart 1244:7 _CustomZone.runUnaryGuarded dart:async/stream_impl.dart 341:11 _BufferingStreamSubscription._sendData dart:async/stream_impl.dart 271:7 _BufferingStreamSubscription._add dart:async/stream_controller.dart 774:19 _SyncStreamControllerDispatch._sendData dart:async/stream_controller.dart 648:7 _StreamController._add dart:async/stream_controller.dart 596:5 _StreamController.add dart:io-patch/socket_patch.dart 2314:41 _Socket._onData dart:async/zone.dart 1442:13 _rootRunUnary dart:async/zone.dart 1335:19 _CustomZone.runUnary dart:async/zone.dart 1244:7 _CustomZone.runUnaryGuarded dart:async/stream_impl.dart 341:11 _BufferingStreamSubscription._sendData dart:async/stream_impl.dart 271:7 _BufferingStreamSubscription._add dart:async/stream_controller.dart 774:19 _SyncStreamControllerDispatch._sendData dart:async/stream_controller.dart 648:7 _StreamController._add dart:async/stream_controller.dart 596:5 _StreamController.add dart:io-patch/socket_patch.dart 1839:33 new _RawSocket. dart:io-patch/socket_patch.dart 1322:14 _NativeSocket.issueReadEvent.issue dart:async/schedule_microtask.dart 40:21 _microtaskLoop dart:async/schedule_microtask.dart 49:5 _startMicrotaskLoop dart:isolate-patch/isolate_patch.dart 122:13 _runPendingImmediateCallback dart:isolate-patch/isolate_patch.dart 193:5 _RawReceivePortImpl._handleMessage C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: The command "setlocal [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -DBUILD_TYPE=Debug -P cmake_install.cmake [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :cmEnd [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :cmErrorLevel [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: exit /b %1 [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :cmDone [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :VCEnd" exited with code 1. [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj] Building Windows application... 00:55 +4 -2: loading C:\Users\johnp\code\samples\testing_app\integration_test\state_mgmt_test.dart [E] Exception: Build process failed. package:flutter_tools/src/base/common.dart 10:3 throwToolExit package:flutter_tools/src/windows/build_windows.dart 311:5 _runBuild ===== asynchronous gap =========================== package:flutter_tools/src/windows/build_windows.dart 77:5 buildWindows ===== asynchronous gap =========================== package:flutter_tools/src/windows/windows_device.dart 62:5 WindowsDevice.buildForDevice ===== asynchronous gap =========================== package:flutter_tools/src/desktop_device.dart 123:7 DesktopDevice.startApp ===== asynchronous gap =========================== package:flutter_tools/src/test/integration_test_device.dart 53:39 IntegrationTestTestDevice.start 00:55 +4 -2: Some tests failed. C:\Users\johnp\code\samples\testing_app> ```
johnpryan commented 2 years ago

It looks like there's some sort of race condition when testing more than one file at a time... I can consistently run flutter test integration_test\app_test.dart -d windows and flutter test integration_test\perf_test.dart -d windows, but not flutter test integration_test -d windows.

dnfield commented 2 years ago

@cbracken might be better set up to help with this (or know who is). My access to Windows machines is fairly limited

cbracken commented 2 years ago

Thanks for the heads-up. Taking a look.

miquelbeltran commented 2 years ago

@csells requested me to put here my findings.

Setup

flutter doctor ``` $ flutter doctor Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel beta, 2.10.0-0.2.pre, on Microsoft Windows [Version 10.0.22000.434], locale en-150) [✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0) [✓] Chrome - develop for the web [✓] Visual Studio - develop for Windows (Visual Studio Community 2019 16.11.4) [✓] Android Studio (version 2020.3) [✓] VS Code (version 1.63.2) [✓] Connected device (3 available) [✓] HTTP Host Availability • No issues found! ```

All tests running on Windows 11 with the Git Bash on Windows Terminal.

Results

Running flutter test integration_test -d windows:

  • the app_test.dart runs and passes
  • the perf_test.dart eventually fails with the following error (see logs)
  • the state_mgmt_test.dart doesn't get to run, also fails (app is a weird state)
logs ``` $ flutter test integration_test -d windows Running "flutter pub get" in testing_app... 2,280ms 00:00 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart B00:26 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart 00:37 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart B00:50 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart Error waiting for a debug connection: The log reader stopped unexpectedly, or never started. 00:50 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart [E] TestDeviceException(Unable to start the app on the device.) package:flutter_tools/src/test/integration_test_device.dart 61:7 IntegrationTestTestDevice.start 00:50 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart B00:50 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart [E] Bad state: Cannot add new events after calling close dart:async/broadcast_stream_controller.dart 243:24 _BroadcastStreamController.add dart:async/zone.dart 1434:47 _rootRunUnary dart:async/zone.dart 1335:19 _CustomZone.runUnary dart:async/zone.dart 1244:7 _CustomZone.runUnaryGuarded dart:async/stream_impl.dart 341:11 _BufferingStreamSubscription._sendData dart:async/stream_impl.dart 271:7 _BufferingStreamSubscription._add dart:async/stream_controller.dart 774:19 _SyncStreamControllerDispatch._sendData dart:async/stream_controller.dart 648:7 _StreamController._add dart:async/stream_controller.dart 596:5 _StreamController.add dart:io-patch/socket_patch.dart 2314:41 _Socket._onData dart:async/zone.dart 1442:13 _rootRunUnary dart:async/zone.dart 1335:19 _CustomZone.runUnary dart:async/zone.dart 1244:7 _CustomZone.runUnaryGuarded dart:async/stream_impl.dart 341:11 _BufferingStreamSubscription._sendData dart:async/stream_impl.dart 271:7 _BufferingStreamSubscription._add dart:async/stream_controller.dart 774:19 _SyncStreamControllerDispatch._sendData dart:async/stream_controller.dart 648:7 _StreamController._add dart:async/stream_controller.dart 596:5 _StreamController.add dart:io-patch/socket_patch.dart 1839:33 new _RawSocket. dart:io-patch/socket_patch.dart 1322:14 _NativeSocket.issueReadEvent.issue dart:async/schedule_microtask.dart 40:21 _microtaskLoop dart:async/schedule_microtask.dart 49:5 _startMicrotaskLoop dart:isolate-patch/isolate_patch.dart 122:13 _runPendingImmediateCallback dart:isolate-patch/isolate_patch.dart 193:5 _RawReceivePortImpl._handleMessage Error waiting for a debug connection: The log reader stopped unexpectedly, or never started. 01:05 +4 -2: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart [E] TestDeviceException(Unable to start the app on the device.) package:flutter_tools/src/test/integration_test_device.dart 61:7 IntegrationTestTestDevice.start 01:05 +4 -2: Some tests failed. ```

Running flutter test integration_test/app_test.dart -d windows:

  • test runs and passes
  • command never finishes (I had to press q to finish it)
logs ``` $ flutter run integration_test/app_test.dart -d windows Launching integration_test/app_test.dart on Windows in debug mode... Building Windows application... flutter: 00:00 +0: Testing App Driver Tests Finding an item in the list Syncing files to device Windows... 53ms Flutter run key commands. r Hot reload. 🔥🔥🔥 R Hot restart. h List all available interactive commands. d Detach (terminate "flutter run" but leave application running). c Clear the screen q Quit (terminate the application on the device). 💪 Running with sound null safety 💪 An Observatory debugger and profiler on Windows is available at: http://127.0.0.1:63630/YJ2FEDpVaPI=/ The Flutter DevTools debugger and profiler on Windows is available at: http://127.0.0.1:9100?uri=http://127.0.0.1:63630/YJ2FEDpVaPI=/ flutter: 00:00 +1: Testing App Driver Tests Testing IconButtons flutter: 00:05 +2: Testing App Driver Tests Verifying whether item gets added to favorites flutter: 00:06 +3: Testing App Driver Tests Testing remove button flutter: 00:10 +4: Testing App Driver Tests (tearDownAll) flutter: Warning: integration_test plugin was not detected. flutter: flutter: If you're running the tests with `flutter drive`, please make sure your tests flutter: are in the `integration_test/` directory of your package and use flutter: `flutter test $path_to_test` to run it instead. flutter: flutter: If you're running the tests with Android instrumentation or XCTest, this means flutter: that you are not capturing test results properly! See the following link for flutter: how to set up the integration_test plugin: flutter: flutter: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab flutter: flutter: 00:10 +5: All tests passed! Application finished. ```

Running performance test:

  • Run the command flutter drive --driver=integration_test/perf_driver.dart --target=integration_test/perf_test.dart -d windows
  • Runs and finishes successfully.
logs ``` $ flutter drive --driver=integration_test/perf_driver.dart --target=integration_test/perf_test.dart -d windows Running "flutter pub get" in testing_app... 990ms Building Windows application... VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:63762/X8EGYWluyoE=/ VMServiceFlutterDriver: Isolate found with number: 706933644293459 VMServiceFlutterDriver: Isolate is paused at start. VMServiceFlutterDriver: Attempting to resume isolate flutter: 00:00 +0: Testing App Performance Tests Scrolling test VMServiceFlutterDriver: Connected to Flutter application. flutter: 00:04 +1: Testing App Performance Tests Favorites operations test flutter: 00:15 +2: Testing App Performance Tests (tearDownAll) flutter: Warning: integration_test plugin was not detected. flutter: flutter: If you're running the tests with `flutter drive`, please make sure your tests flutter: are in the `integration_test/` directory of your package and use flutter: `flutter test $path_to_test` to run it instead. flutter: flutter: If you're running the tests with Android instrumentation or XCTest, this means flutter: that you are not capturing test results properly! See the following link for flutter: how to set up the integration_test plugin: flutter: flutter: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab flutter: flutter: 00:15 +3: All tests passed! All tests passed. ```

Running state_mgmt_test.dart:

  • Same result as with the app_test.dart:
  • Runs successfuly
  • the testing app remains open, needs to be closed with q
logs ``` $ flutter run integration_test/state_mgmt_test.dart -d windows Launching integration_test/state_mgmt_test.dart on Windows in debug mode... Building Windows application... flutter: 00:00 +0: Testing App State Management Tests Verifying add method Syncing files to device Windows... 55ms Flutter run key commands. r Hot reload. 🔥🔥🔥 R Hot restart. h List all available interactive commands. d Detach (terminate "flutter run" but leave application running). c Clear the screen q Quit (terminate the application on the device). 💪 Running with sound null safety 💪 An Observatory debugger and profiler on Windows is available at: http://127.0.0.1:63842/agkZOurH5FE=/ The Flutter DevTools debugger and profiler on Windows is available at: http://127.0.0.1:9100?uri=http://127.0.0.1:63842/agkZOurH5FE=/ flutter: 00:00 +1: Testing App State Management Tests Verifying remove method flutter: 00:00 +2: Testing App State Management Tests (tearDownAll) flutter: Warning: integration_test plugin was not detected. flutter: flutter: If you're running the tests with `flutter drive`, please make sure your tests flutter: are in the `integration_test/` directory of your package and use flutter: `flutter test $path_to_test` to run it instead. flutter: flutter: If you're running the tests with Android instrumentation or XCTest, this means flutter: that you are not capturing test results properly! See the following link for flutter: how to set up the integration_test plugin: flutter: flutter: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab flutter: flutter: 00:00 +3: All tests passed! Application finished. ```

Removing the files drive.dart perf_driver.dart and perf_test.dart from the integration_test folder, then running flutter test integration_test -d windows:

  • The test app_test.dart runs and passes.
  • The test state_mgmt_test.dart starts and fails.
  • A blank app window appears on the screen and remains open.
logs ``` $ flutter run integration_test -d windows Target file "integration_test" not found. Miquel@MIGUEL-WINDOWS MINGW64 ~/Documents/dev/samples/testing_app (master) $ flutter test integration_test -d windows 00:00 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart B00:13 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart 00:24 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart B00:37 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart Error waiting for a debug connection: The log reader stopped unexpectedly, or never started. 00:37 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart [E] TestDeviceException(Unable to start the app on the device.) package:flutter_tools/src/test/integration_test_device.dart 61:7 IntegrationTestTestDevice.start 00:37 +4 -1: Some tests failed. ```

Summary

  • It seems that when more than one test file is in the integration_test folder, the command flutter test integration_test will fail on the second file.
  • Running a single test leaves the command running and needs to be closed with q (or control+c).
  • The fact that the performance tests are in the same folder seems unrelated to the root cause of this issue.