search

eBay / flutter_glove_box

Various eBay tools for Flutter development
BSD 3-Clause "New" or "Revised" License
323 stars 69 forks source link

[Feature Request] Ability to run tests sequentially (scenarios AND devices) #181

Open mstagitis opened 1 year ago

mstagitis commented 1 year ago

Hello all - First off, thanks for the library! It's great.

I have a FR to allow the tests to run sequentially across test cases and devices. In the case where we have 2 scenarios and 2 devices, we should have 4 tests run sequentially.

I'm in need of this due to some limitations of test libraries/dependencies that I'm using. Thanks!

mstagitis commented 1 year ago

A follow-up here: It looks like there is state leaking between test cases when using GoldenToolkit. When I only run a single scenario for a single device, my test passes. Once I add an additional scenario or a second device, the tests fail with the following error:

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown running a test:
setState() or markNeedsBuild() called during build.
This SearchScreen widget cannot be marked as needing to build because the framework is already in
the process of building widgets. A widget can be marked as needing to be built during the build
phase only if one of its ancestors is currently building. This exception is allowed because the
framework builds parent widgets before children, which means a dirty descendant will always be
built. Otherwise, the framework might not visit this widget during this build phase.
The widget on which setState() or markNeedsBuild() was called was:
  SearchScreen
The widget which was currently being built when the offending call was made was:
  UncontrolledProviderScope

I'm using RiverPod to maintain state. In a nutshell, the test case updates a query field, which triggers a new provider value and therefore tries to rebuild the SearchScreen (mentioned in the error above). My guess is that the SearchScreen for the second device is already being built, so this error is thrown. But shouldn't these two tests have totally disjoint widget/state trees?

Note: I am able to run end-to-end tests with no issues and cannot reproduce this with the app running locally; the issue exists solely in these tests.