Run e2e test on CI - Githubissues

This proved more difficult than expected, because currently the app does not run on x86_64 architecture (https://github.com/digidem/comapeo-mobile/issues/446), and most CI only supports emulators running on x86_64.

Some options evaluated by @cimigree and myself:

Run emulator on Apple Silicon runners on Github

It seems like Apple Silicon runners on Github do not have hardware emulation support (Apple Intel runners do), which prohibits running the Android emulator on the runner.
Run emulator on new Github large Arm Ubuntu runners (paid)

It is unclear whether hardware emulation is supported on these runners, but also, Android SDK tools do not seem to currently support Arm on Linux, only Apple Silicon.
Run tests on Genymotion emulator using their Saas product

Genymotion Saas does not support arm64 images
Run tests on Genymotion emulator using their Device Image product

This could potentially work, but it requires setting up machines on AWS, GCP or MS Azure. The overhead of maintaining both the machines and the Genymotion config could be a lot.
Run tests on "bare" AWS machines

This Gist outlines a way to configure an AWS EC2 Arm machine to run an emulator. This route may work, but requires diving into the complexities of running AWS EC2 images.
Run tests on Maestro Cloud

Maestro cloud only supports x86_64 architecture.
Run tests on real devices using BrowserStack

BrowserStack does not support running tests with Maestro, but it does have beta support for Detox.

Many of these options just will not work, and none of them are ideal, and all the options that could work require an unknown amount of effort to get working.

Out of not-great options, it seems like BrowserStack is the best option, particularly as they have a very generous Open Source plan that offers free unlimited real-device testing (5 parallel devices). This does require re-writing tests in Detox instead of Maestro. The negative is that Detox support in BrowserStack is still in Beta, so we could encounter issues. A positive (a major one in some respects) is that running tests on real devices is more likely to identify issues than tests on emulators or on a different architecture (e.g. most user devices are Arm, vs. x86_64 emulator). Our user-testing to date has already identified issues that arise from specific Arm chips, or differences between Arm7 and Arm64. Testing on BrowserStack would catch this kind of issue, if the device that causes the issue is available on their infrastructure.

I have created an orphaned branch of this repo (so that it is tied to this open source project in BrowserStack, but the code is just a template app) that runs tests in BrowserStack, either from a local CLI command, or via a Github Action. There is just one Detox test to check the main screen of the app loads. It seems to work ok:

For now running e2e tests in CI is lower priority than MVP issues, so we can address this post-MVP:

Copy code from the browserstack-testing branch into a new branch from develop.
Check a basic detox test (checking first screen appears) works locally and on BrowserStack
Re-write tests from Maestro to Detox

(3) can be done incrementally - we can continue to run Maestro tests locally until they are all migrated to Detox.

digidem / comapeo-mobile

Run e2e test on CI #400

To Do

To note