Add ability to run `flutter driver` tests on multiple devices at the same time

[jorgecoca]

Use case

As a developer, I would like to be able to run the same suite of integration tests on all the devices connected to my machine. This is will allow me to test my application in many different devices in parallel, being able to obtain results faster.

Proposal

There could be two different proposals:

• Allow multiple instances of flutter drive to run at the same time, so I could run the following commands without getting the error Waiting for another flutter command to release the startup lock...
  • Terminal1: flutter drive ... -d iphone
  • Termina2: flutter drive ... -d pixel
• Allow the -d flag (devices) to pass the value all, running integration tests in all the devices connected to that computer: flutter drive ... -d all

[xster]

@jonahwilliams @dnfield I vaguely remember there were discussions of multiple simultaneous flutter tool runs. Is it just a matter of moving the bootstrapping part to be a separate atomic operation vs the rest of the normal Dart snapshot part of the tool?

[mmcc007]

In the meantime, it is possible to automatically run integration tests in series on real devices, emulators and simulators (while optionally collecting screenshots) using https://github.com/mmcc007/screenshots and in parallel on a large number of real devices on AWS Device Farm using: https://github.com/mmcc007/sylph

[Hixie]

This would imply running the host side multiple times, one per client.

[jonahwilliams]

Allow multiple instances of flutter drive to run at the same time, so I could run the following commands without getting the error Waiting for another flutter command to release the startup lock...

This is required because flutter drive will first build the application if it doesn't already exist. You can use a prebuilt or already running application to work around this - though the lock will be released once the build is finished.

I think it would be possible to do this with only small changes to the Flutter tool. They way to go about doing so would be by running flutter run --mode --start-paused -t entrypoint once per device target you wanted to run. Then, for each device target you can separately run flutter driver --use-existing-app=observatory.

The caveat would be that I don't think we make it easy to get the observatory uri such that chaining these commands together would be easy - that is something I plan to look into.

@jorgecoca do you think this would solve your immediate problem?

[jorgecoca]

Yes, I think that should solve the problem, or at least cut some execution time. Thanks!

[jonahwilliams]

@jorgecoca I opened a PR with a small set of changes and example workflow in https://github.com/flutter/flutter/pull/41583 . When you have a chance, would you try it out and see if it works for you?

[jorgecoca]

My apologies, I forgot to reply, but I tried this a while ago actually 🤦‍♂I have seen an improvement definitely, and I was able to run tests in both an iOS simulator and an Android emulator. Thank you so much! 💯

[jonahwilliams]

Thank you, I'll work on getting this landed!

[neiljaywarner]

@Hixie @jonahwilliams @jorgecoca Can someone please update this page for people who land here searching for this with a "where this ended up", ideally with a link to some documentation for "official" guidance? This thread makes it seem like there might be two options, one old one new, etc.

thanks so much

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.