search

peartherapeutics / bitrise-aws-device-farm-runner

Custom Bitrise step to run tests on aws device farm
MIT License
10 stars 21 forks source link

Wait for Device Farm builds to complete & return results #15

Closed jcarr-sailthru closed 7 years ago

jcarr-sailthru commented 7 years ago

This PR ensures the step waits until the run completes or fails in DeviceFarm, rather than simply scheduling and returning success. This can take a while depending on the number/type of tests and the number of devices.

We export the build results, both the raw JSON response from AWS as well as a nice human-friendly text string. As these are exports, they can be used by later stages in the Bitrise build process, eg feeding into a Slack step to return more information about how the build went.

Note that currently this does not include a break down of each individual job inside the build run. This could be a useful extension ontop of this work in future.

jcarr-sailthru commented 7 years ago

Looks like something has gone wrong with whitespacing... will fix and amend PR, but please feel free to send through feedback meanwhile.

jcarr-sailthru commented 7 years ago

hi @fadookie - please take a look now and let me know what you think.

fadookie commented 7 years ago

@jcarr-sailthru I'm doing some final testing on my end and then I will cut a new step release. I noticed that due to this new blocking behavior, when doing a crossplatform test run, the android run doesn't get kicked off until after the ios run completes. If you have time it would be great if these runs could be started and monitored in parallel, it should halve the polling time for crossplatform builds.

fadookie commented 7 years ago

@jcarr-sailthru This would also address another issue I just noticed which is that if the ios run fails, the android run won't even be started which might not be desirable. In any case I will cut a new release as I think many people could benefit from the new functionality as-is.

jcarr-sailthru commented 7 years ago

thanks for merging @fadookie.

The parallelism will be a bit tricky, it might be required to consider moving parts of the script to a language that can do some kind of threading, eg Python or Ruby rather than trying to hack it into bash with background processes.

We aren't doing cross platform builds here so probably not something I'll end up making time to work on.

fadookie commented 7 years ago

@jcarr-sailthru yeah, if we wind up switching to polling I'll have a crack at this myself. Thanks again!