Solve issue with Nightly E2E tests ignoring Sauce Labs limits

[jakubgs]

Currently we have only 1 build session with 8 devices(which is silly slow), and this limit is enforced simply by Jenkins job config:

https://ci.status.im/job/status-mobile/job/e2e/job/status-app-prs/configure

This works well enough for our development instance of Jenkins at https://ci.status.im/, but it does not take into account our separate release instance which runs mobile nightly builds which also trigger end-to-end tests, except no limit is enforced due to it being a separate Jenkins instance.

This result is that the E2E job fails with:

ERROR medium/test_activity_center.py::TestActivityMultipleDevicePR::test_navigation_jump_to@new_one_3
  - AttributeError: '_asyncio.Future' object has no attribute 'session_id'

https://ci.status.im/job/status-mobile/job/e2e/job/status-app-prs-rerun/2750/console

There are at least two possible solutions:

1. Run E2E for nightly on the devel CI by fetching nightly artifacts directly from DigitalOcean bucket.
2. Run E2E for nightly on the release CI but implement some kind of global session locking mechanism.

The former is simpler, but does require extra logic to find the latest nightly build. The latter is the more correct solution, but would require finding or developing some kind of locking API to use in both jobs. One possibility would be to use Consul API locking mechanism in some kind of script.

[yakimant]

@jakubgs, how can I find a build, which was running at the same time to https://ci.status.im/job/status-mobile/job/e2e/job/status-app-prs-rerun/2750?

[jakubgs]

Good question actually. I actually can't find a matching nightly e2e:

• https://ci.infra.status.im/job/status-mobile/job/e2e/job/status-app-nightly/2615/

So maybe this one failed for some other reason and the nightly e2e is a red herring in that case.

[yakimant]

Yes, it failed because of a different reason:

• e2e jenkins run was manually cancelled, leaving saucelabs build running
• it was blocking the queue and tests were failing

nightly e2e usually interfere the pr e2e when run in working hours. For example due to nightly build issue fixed in the morning

[yakimant]

For TLDR - jump to Proposal. Internsion of this comment is to present rather than read.

UPDATE 1: Applied changes after review with @jakubgs

Options

Option 1: Switch e2e/nightly from ci.infra to ci. Keep using platform/android-e2e results from ci.infra.

• No build dependency
• Artifact dependency: take latest from same day or fail

Option 2: Switch e2e/nightly from ci.infra to ci. Use results from platform/android-e2e from ci.

• No build dependency
• Artifact dependency: take latest on same branch

Option 3: Same as 2, but run platform/android-e2e build inside e2e/nightly run with build step.

• Build dependency
• Artifact dependency

Option 4: Same as 2, but add both platform/android-e2e and e2e/nightly as build dependencies to meta runner, eg e2e-nightly

• Build dependency
• Artifact dependency

Build dependency

• 1:1 commit hash map for test and build in sync
• will require to build every time

Solution

Option 2 + Option 4 combined:

• If you run e2e/nightly - it will not trigger a build, but will use previous build result
• If you run e2e-nightly - it will run a build and e2e tests

Proposal

• Meta runner will trigger both platform/android-e2e and e2e/nightly every night on ci (same as on ci.infra before)
• New Jenkins.e2e-nightly
• Change BUILD_TYPE to e2e, because you can't run nightly on ci
• Cleanup:
  • ci.infra/e2e
  • ci/e2e other than prs, prs-rerun, upgrade and nightly
  • Jenkins.combined to remove e2e tests and platform/android-e2e

Things to decide

• Which build type to use for nightly e2e instead of nightly. See more in comparison sheet Settled to e2e
• Name of a new meta runner (or use the manual) one. Nightly is a bit confusing with different build type. Settled to e2e-nightly

Future

• Investigate switch all to multitrigger, remove platform/pr separation
• nightly/release make no sense on ci, e2e/manual/pr make no sense on ci.infra. Investigate limiting the build types.
• Investigate on publish parameter, worth to remove on manual?
• e2e ios doesn't build

[jakubgs]

I think think this is mostly correct except for:

• Using ci/Jenkinsfile.combined for this is adding unnecessary complexity. It's better to just add something like ci/tests/Jenkinsfile.e2e-nightly-starter that will call ci/Jenkinsfile.android and ci/tests/Jenkinsfile.e2e-nightly.
• Build type probably doesn't matter since architectured depends on the ABI parameter, and we don't care about signature.
• We should probably just put the meta job on ci.status.im under status-mobile/e2e-nightly for visibility.

Good research.

[yakimant]

@jakubgs I agree, ammended the proposal

[yakimant]

@churik, please read the proposal part and let's review tomorrow. The main part if you are ok to switch nightly e2e from nightly build type to e2e build type. The diff is here: https://docs.google.com/spreadsheets/d/1vigaW2yFpQoS5xq-p1pN_dx2q_O4hfCVK5awrl9snXg/edit#gid=143162119

[churik]

@jakubgs @yakimant thank you for the detailed explanation! It is OK from e2e perspective to change type of the build, we can use this approach - there are no dependencies on the type of the build currently.

[yakimant]

Still need testing and cleanup, reopening.

[yakimant]

@churik, nightly e2e are running on ci.status.im now, please have a look: https://ci.status.im/job/status-mobile/job/e2e-nightly/

I'm going to cleanup e2e from ci.infra.status.im now.

[churik]

yes, thank you for your work!!!

[yakimant]

Removed:

• ci.infra
  • e2e
  • platform/android-e2e
• ci / e2e
  • status-api-tests
  • status-app-performance
  • test-copy-apk-status-apps-prs

Backup is in restic for 30 days just in case.

[yakimant]

Both ci.infra/nightly and ci/e2e-nightly work well. e2e-nightly and regular e2e will queue for saucelabs resources now.

[yakimant]

Another issue appeared while discussing the nightlies, I created a ticket for it: https://github.com/status-im/status-mobile/issues/16966