search

flutter / devtools

Performance tools for Flutter
https://flutter.dev/docs/development/tools/devtools/
BSD 3-Clause "New" or "Revised" License
1.55k stars 319 forks source link

debugger_evaluation_test is flaky #7099

Open whesse opened 7 months ago

whesse commented 7 months ago

The test "returns privates only from library" in packages/devtools_app/test/debugger/debugger_evaluation_test.dart is failing flakily on the builder https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/flutter-linux-customer-testing

It is only this test that is failing, always with the same stack trace, and it is happening more than 10% of the time. The stack trace says that the test is failing and always has the message: "This test failed after it had already completed. | Make sure to use a matching library which informs the test runner | of pending async work."

This flakiness is affecting our CI, so if the problem with the test can't be fixed easily, it would be good to disable it until it can be fixed.

The full stack trace from the logs is:

02:21 +566 ~8 -1: /b/s/w/ir/x/t/flutter_customer_testing.flutter_devtools.MFWIAI/tests/packages/devtools_app/test/debugger/debugger_evaluation_test.dart: EvalOnDartLibrary returns privates only from library [E]
|   [Sentinel kind: Collected, valueAsString: <collected>] from getObject()
|   package:vm_service/src/vm_service.dart 265:34                              new _OutstandingRequest
|   package:vm_service/src/vm_service.dart 1771:25                             VmService._call.<fn>
|   package:vm_service/src/vm_service.dart 1783:8                              VmService._call
|   package:vm_service/src/vm_service.dart 1003:7                              VmService.getObject
|   package:devtools_app/src/service/vm_service_wrapper.dart 164:18            VmServiceWrapper.getObject
|   package:devtools_app/src/shared/scripts/script_manager.dart 125:10         _ScriptCache.getScript
|   package:devtools_app/src/shared/scripts/script_manager.dart 77:25          ScriptManager.getScript
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 518:40  DebuggerController._createStackFrameWithLocation
|   dart:async                                                                 Future.wait
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 461:20  DebuggerController._getStackInfo
|   ===== asynchronous gap ===========================
|   dart:async                                                                 _Completer.completeError
|   package:vm_service/src/vm_service.dart 272:18                              _OutstandingRequest.completeError
|   package:vm_service/src/vm_service.dart 1860:17                             VmService._processResponse
|   package:vm_service/src/vm_service.dart 1840:9                              VmService._processMessageStr
|   package:vm_service/src/vm_service.dart 1798:7                              VmService._processMessage
|
|   This test failed after it had already completed.
|   Make sure to use a matching library which informs the test runner
|   of pending async work.
|   package:vm_service/src/vm_service.dart 265:34                              new _OutstandingRequest
|   package:vm_service/src/vm_service.dart 1771:25                             VmService._call.<fn>
|   package:vm_service/src/vm_service.dart 1783:8                              VmService._call
|   package:vm_service/src/vm_service.dart 1003:7                              VmService.getObject
|   package:devtools_app/src/service/vm_service_wrapper.dart 164:18            VmServiceWrapper.getObject
|   package:devtools_app/src/shared/scripts/script_manager.dart 125:10         _ScriptCache.getScript
|   package:devtools_app/src/shared/scripts/script_manager.dart 77:25          ScriptManager.getScript
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 518:40  DebuggerController._createStackFrameWithLocation
|   dart:async                                                                 Future.wait
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 461:20  DebuggerController._getStackInfo
|   ===== asynchronous gap ===========================
|   dart:async                                                                 _Completer.completeError
|   package:vm_service/src/vm_service.dart 272:18                              _OutstandingRequest.completeError
|   package:vm_service/src/vm_service.dart 1860:17                             VmService._processResponse
|   package:vm_service/src/vm_service.dart 1840:9                              VmService._processMessageStr
|   package:vm_service/src/vm_service.dart 1798:7                              VmService._processMessage
|
|   [Sentinel kind: Collected, valueAsString: <collected>] from getObject()
|   package:vm_service/src/vm_service.dart 265:34                              new _OutstandingRequest
|   package:vm_service/src/vm_service.dart 1771:25                             VmService._call.<fn>
|   package:vm_service/src/vm_service.dart 1783:8                              VmService._call
|   package:vm_service/src/vm_service.dart 1003:7                              VmService.getObject
|   package:devtools_app/src/service/vm_service_wrapper.dart 164:18            VmServiceWrapper.getObject
|   package:devtools_app/src/shared/scripts/script_manager.dart 125:10         _ScriptCache.getScript
|   package:devtools_app/src/shared/scripts/script_manager.dart 77:25          ScriptManager.getScript
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 518:40  DebuggerController._createStackFrameWithLocation
|   dart:async                                                                 Future.wait
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 461:20  DebuggerController._getStackInfo
|   ===== asynchronous gap ===========================
|   dart:async                                                                 _Completer.completeError
|   package:async/src/cancelable_operation.dart 428:23                         CancelableCompleter.complete.<fn>
|   ===== asynchronous gap ===========================
|   dart:async                                                                 Future.then
|   package:async/src/cancelable_operation.dart 425:11                         CancelableCompleter.complete
|   package:async/src/cancelable_operation.dart 34:52                          new CancelableOperation.fromFuture
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 482:46  DebuggerController._getFullStack
|   ===== asynchronous gap ===========================
|   dart:async                                                                 _CustomZone.registerUnaryCallback
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 481:5   DebuggerController._getFullStack
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 444:17  DebuggerController._pause
|   ===== asynchronous gap ===========================
|   dart:async                                                                 _CustomZone.registerUnaryCallback
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 378:5   DebuggerController._pause
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 303:19  DebuggerController._handleDebugEvent
|   ===== asynchronous gap ===========================
|   dart:async                                                                 _StreamImpl.listen
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 124:29  DebuggerController._initialize
|   package:devtools_app/src/screens/debugger/debugger_controller.dart 56:7    new DebuggerController
|   test/debugger/debugger_evaluation_test.dart 39:26                          main.<fn>
|   ===== asynchronous gap ===========================
|   dart:async                                                                 _CustomZone.registerUnaryCallback
|   test/debugger/debugger_evaluation_test.dart 38:5                           main.<fn>
kenzieschmoll commented 7 months ago

@elliette can you take a look at this debugger test flake?

whesse commented 7 months ago

The test is still failing on our builder, flakily, after we resume running it. The error is the same.

It could be that something is throwing an exception in the flutter code that is executed after the test runs, so that the async expect in the test is completing correctly, and waiting for the things it needs to, but there is still more happening in the app (maybe as part of shutdown?).

Builder that is failing is https://ci.chromium.org/ui/p/dart/builders/ci.sandbox/flutter-linux-customer-testing/467/overview

Is there something that can be done to get more useful stack traces here. The stack trace is exactly the same as before. Shall we roll back the devtools tests in the customer test registry again? I think so.

kenzieschmoll commented 6 months ago

Skipped the flaky test until we have a fix: https://github.com/flutter/tests/pull/344