search

mmcc007 / sylph

Runs Flutter integration tests on real devices in cloud.
GNU General Public License v3.0
157 stars 43 forks source link

Restarting test when devices are available #102

Open fvisticot opened 4 years ago

fvisticot commented 4 years ago

Is there a solution to restart the test once the devices are available on the Device farm side ? Some time I get the error: Error: device: 'name:Apple iPhone 8 Plus, model:A1864, os:12.1.0, deviceType:ios, formFactor:phone, availability:BUSY' is busy.

mmcc007 commented 4 years ago

If in a CI/CD environment can perhaps auto-retry. Other than that the only options AFAIK are to wait for device to become available or change to a less popular device.

tim-fdc commented 4 years ago

Hi, having a fork here that skips busy devices. Still need to finalize the changes. Also added a random selection of pools. Both is via the sylph.yaml. For us this really helped to get more reliable test executions and good device coverage. Oh and yes, agree with @mmcc007 regarding the rety. We have that as well.

So, summarizing:

@mmcc007 Regarding the local dependencies we also added a check to avoid adding packages, which may lead to cyclic dependencies. Would you be interested in a PR?

Happpy Hakking Tim

mmcc007 commented 4 years ago

@tim-fdc Sure, would love to see a PR on this. Thnx.

mcmarkj commented 4 years ago

@tim-fdc Are you able to share your fork, please? Would be incredibly useful!

mcmarkj commented 4 years ago

I gave it a stab: https://github.com/mmcc007/sylph/pull/109

We were facing lots and lots of issues where builds would fail very often as devices were in use. Instead we opted to have a big list of devices to test against, confident this covers the devices in our user base. This means that if any devices were unavailable this was acceptable. But if every device was unavailable that should == a failed step in our CI pipeline.

I've not gone for the full re-try as discussed above. Quite simply because we use GitHub actions and if we waited until a device became available I'd bankrupt the company. Instead, I've set it to skip but it would fail if no devices are available.

Perhaps a further improvement that someone could look into is a toggle on what action you'd like when a device is unavailable. I.e. die if any one device is unavailable / skip unavailable devices and continue with tests.