CI: Appium tests on Sauce all failing

janpio commented 5 years ago

All failing:

iOS: https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514197 https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514198 https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514199 https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514200

Android: https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514201 https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514202 https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514203 https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514204 https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514205 https://travis-ci.org/apache/cordova-plugin-camera/jobs/530514206

Problem is that Paramedic itself doesn't have any tests of this kind, so it is difficult to test here in this repo first :/

janpio commented 5 years ago

I checked older, successful CI runs of cordova-plugin-camera, and it seems the Appium tests haven't been run for quite some time:

...
cordova-paramedic: tests have been completed
cordova-paramedic: Tests finished
Start running Appium tests...
Running Appium tests on Sauce Labs
Couldn't find Appium tests, skipping...
Completed tests at 5:21:17 PM
Getting saucelabs jobs details...
...

https://travis-ci.org/apache/cordova-plugin-camera/builds/528500727?utm_source=github_status&utm_medium=notification

Somehow "Couldn't find Appium tests, skipping..." got fixed in the last 5 days.

janpio commented 5 years ago

Could find a successful CI run with Appium tests actually being executed for the contacts plugin: https://travis-ci.org/apache/cordova-plugin-contacts/jobs/491749438 3 months ago.

On further looking:

A Android run for the camera plugin that succeeded: https://travis-ci.org/apache/cordova-plugin-camera/jobs/514613162 ~1 month ago

Before that, ~2 months ago, we have successful Appium runs for iOS and Android: https://travis-ci.org/apache/cordova-plugin-camera/jobs/507356980 https://travis-ci.org/apache/cordova-plugin-camera/jobs/507356981 https://travis-ci.org/apache/cordova-plugin-camera/jobs/507356982 https://travis-ci.org/apache/cordova-plugin-camera/jobs/507356983 https://travis-ci.org/apache/cordova-plugin-camera/jobs/507356984

So the question now is: What changed between these successful runs and now?

janpio commented 5 years ago

We changed the OS versions the tests are run on (all new iOS versions as old simulators are gone)
We added an environment for the tests to run on (local instead of just saucelabs)
We changed the whole travis.yml structure (but this shouldn't influence things)
We changed the OS version configuration details (saucelabs parameters like appiumVersion etc)
We rewrote good parts of paramedic, that is used to actually run the tests

janpio commented 5 years ago

Similar OS/version combinations being run then and now are: Andorid 4.4, 5.1, 6.0, 7.0. For these combinations debugging makes most sense, as theoretically it should still work.

janpio commented 5 years ago

Looking into the failures we are getting a bit, they look different at first, but turn out to be the same thing:

iOS:

- Failed: Failed to start a session

Android:

- ReferenceError: lastFailureReason is not defined

Turns out the Android error comes from this line: https://github.com/apache/cordova-plugin-camera/blob/295e928784e2a9785982fbf15b05215a317774df/appium-tests/android/android.spec.js#L353

Here lastFailureReason is indeed undefined, it doesn't appear anywhere else. Otherwise this would also output:

Failed to start a session

Why though... no idea yet.

janpio commented 5 years ago

Also tracked at: https://github.com/apache/cordova-plugin-camera/issues/466 https://github.com/apache/cordova-plugin-camera/issues/465

janpio commented 5 years ago

As a temporary workaround, the Appium tests were removed from camera. A new PR will add them back: https://github.com/apache/cordova-plugin-camera/pull/469

apache / cordova-paramedic

CI: Appium tests on Sauce all failing #145