test_driver/scroll_perf.dart is flaky

[Hixie]

I ran it three times. One time it said it timed out waiting for the app. A second time it worked fine. A third time it said:

$ flutter drive --profile test_driver/scroll_perf.dart
Using device Nexus 6.
Starting application: lib/main.dart
Building APK in profile mode (android-arm-profile)...
Warning: signing the APK using the debug keystore.
Built build/app.apk (12.7MB).
Observatory listening on http://127.0.0.1:8183
00:00 +0: scrolling performance test (setUpAll)
[info ] FlutterDriver: Connecting to Flutter application at http://localhost:8183
[trace] FlutterDriver: Looking for the isolate
00:00 +0 -1: scrolling performance test (setUpAll)
  Bad state: No element
  dart:core                                               List.first
  package:flutter_driver/src/driver.dart 85:43            FlutterDriver.connect.<async>
  dart:async                                              _SyncCompleter.complete
  package:vm_service_client/vm_service_client.dart 207:7  VMServiceClient.getVM.<async>
  dart:async                                              _SyncCompleter.complete
  package:json_rpc_2/src/client.dart 175:25               Client._handleSingleResponse
  package:json_rpc_2/src/client.dart 165:7                Client._handleResponse
  dart:async                                              _StreamController.add
  package:json_rpc_2/src/peer.dart 92:36                  Peer.listen.<fn>
  ===== asynchronous gap ===========================
  dart:async                                              _Completer.completeError
  package:flutter_driver/src/driver.dart 188:3            FlutterDriver.connect.<async>
  dart:async                                              _SyncCompleter.complete
  package:vm_service_client/vm_service_client.dart 207:7  VMServiceClient.getVM.<async>
  dart:async                                              _SyncCompleter.complete
  package:json_rpc_2/src/client.dart 175:25               Client._handleSingleResponse
  package:json_rpc_2/src/client.dart 165:7                Client._handleResponse
  dart:async                                              _StreamController.add
  package:json_rpc_2/src/peer.dart 92:36                  Peer.listen.<fn>
  ===== asynchronous gap ===========================
  dart:async                                              _asyncThenWrapperHelper
  package:flutter_driver/src/driver.dart                  FlutterDriver.connect
  test_driver/scroll_perf_test.dart 14:36                 main.<fn>.<fn>.<async>
  ===== asynchronous gap ===========================
  dart:async                                              Future.Future.microtask
  test_driver/scroll_perf_test.dart                       main.<fn>.<fn>

00:00 +0 -1: scrolling performance test (tearDownAll)
00:00 +0 -1: Some tests failed.

Unhandled exception:
Dummy exception to set exit code.
#0      _rootHandleUncaughtError.<anonymous closure> (dart:async/zone.dart:1138)
#1      _microtaskLoop (dart:async/schedule_microtask.dart:41)
#2      _startMicrotaskLoop (dart:async/schedule_microtask.dart:50)
#3      _Timer._runTimers (dart:isolate-patch/timer_impl.dart:394)
#4      _Timer._handleMessage (dart:isolate-patch/timer_impl.dart:414)
#5      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:148)
Stopping application instance.

cc @yjbanov

[eseidelGoogle]

Should this be in "Today"?

[yjbanov]

@Hixie which scroll_perf.dart is this referring to? We have a few (1, 2, 3). We only run complex_layout/test_driver/scroll_perf.dart on the devicelab on a regular basis, and the build history looks very stable. If the flaky test is one we're not running regularly, we should probably remove it as it's probably bitrotten by now.

[Hixie]

This was referring to "Scroll perf (complex_layout)" on our dashboard. It's in "Today" because it means the dashboard is broken and we are unable to notice when we regress performance.

[yjbanov]

I see that we are failing to record the frames (or perhaps there really is only one frame). We should definitely fix that. However, I'm not seeing flakiness or the stacktrace that looks like the one in the original bug report.

How about we close this issue (unless anyone can still reproduce it) and open a new one to fix the missing frame data?

[Hixie]

So long as the dashboard gets fixed, I don't mind what bug tracks it. :-)

We should probably also have a bug to track the flakiness of the benchmark when run locally, though.

[eseidelGoogle]

This has made no movement in "Today" for 25 days, that doesn't seem right. Removing the milestone.

[eseidelGoogle]

@goderbauer fixed a bunch of flakes last month. Unclear if that affected this?

[goderbauer]

On the dashboard, this test is passing every single run.

[zanderso]

This issue appears stale. Closing.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.