search

awslabs / aws-device-farm-gradle-plugin

Gradle plugin for AWS Device Farm.
Apache License 2.0
61 stars 32 forks source link

Test run only gets enqueued, plugin doesn't wait for tests to finish #11

Closed kkozmic-seek closed 6 years ago

kkozmic-seek commented 8 years ago

Running version 1.2 of the plugin I was trying to execute some Instrumentation (Espresso) tests.

There are two things that happen that I think are wrong.

  1. The build finishes very quickly with a link to the device farm console, where all devices show up as unavailable. If I run the tests through the Device Farm UI on the web (same tests, same device pool, same apk) it all works fine screen shot 2016-01-13 at 3 21 39 pm
  2. The build is reported as successfully passing, which it obviously is not. Should be reported as failure.
kkozmic-seek commented 8 years ago

The issue 1) turns out was due to useUnmeteredDevices() which came from the documentation. Removing it fixes the problem (perhaps some clearer exception message would help).

Issue 2) still exists - the plugin does not wait for the tests to complete, just kicks them off which makes it somewhat useless in scenarios we were most keen to utilise it - on our CI server (TeamCity)

bradley-curran commented 6 years ago

Thanks for reporting the behavior that causes tests to be skipped.

I've changed the documentation to try and help others avoid this issue. If it keeps coming up I'll look into what else we can do to avoid this issue.