search

TryQuiet / quiet

A private, p2p alternative to Slack and Discord built on Tor & IPFS
https://www.tryquiet.org
GNU General Public License v3.0
1.93k stars 82 forks source link

Setup GitHub runner for Detox tests (Apple Silicon) #1879

Open siepra opened 11 months ago

siepra commented 11 months ago

UPDATE: Whoever takes on this ticket will either set up the existing runner on a Quiet-controlled cloud VPN and express mail it to the US, or they will migrate the tests to Github M1 runners, whichever is easier (preference for Github).

Currently tests fail sometimes because of Metro caching something, and we've had problems before of false positive passing tests due to leftover files from the previous test run. This is one reason to prefer running the tests on GH runners.

Including previous notes below

Here are the milestones for the full completion of mobile testing automation process:

NOTE: Before setting up visual regression tests, we should try this one https://github.com/TryQuiet/quiet/issues/1874

=======

As Bitrise turned our not to be a perfect fit for running automated e2e Detox tests, we're deciding to setup our own GitHub runner.

We need a machine that runs on Apple Silicon (arm64) and is capable of swiftly building React Native applications. I think at least 32GB RAM and 500GB storage space is rational

siepra commented 11 months ago

We're waiting for the machine to be delivered to the office in Kraków

holmesworcester commented 11 months ago

I created #1915 after a friend recommended Amazon Device Farm for testing Android compatibility on real devices (not simulator.) The pricing seems reasonable. Is this an option?

siepra commented 11 months ago

Definitely worth trying

siepra commented 10 months ago

I'm moving to blocked as we must update Tor on mobile first

holmesworcester commented 10 months ago

I'm moving to blocked as we must update Tor on mobile first

Can you say a bit more about why this is blocked by updating Tor?

holmesworcester commented 5 months ago

As part of this work, let's document any place where the Detox tests diverge from how the app would naturally run in production.

Of the top of my head, the APK vs. APKS is one thing, and then there's something siepra mentioned about the React Native Bridge and websockets.

siepra commented 4 months ago

As part of this work, let's document any place where the Detox tests diverge from how the app would naturally run in production

I think Detox is self-explanatory about that https://wix.github.io/Detox/

Detox puts the actual build under test so the factors to consider are:

I don't know what the consequences of this are, though.

We currently put debug builds under test (debuggable, less setup complexity, probably faster), but we can do production builds if needed.

of the top of my head, the APK vs. APKS is one thing

I don't get it, can you expand?

there's something siepra mentioned about the React Native Bridge and websockets

I sure mentioned it but not in the context of Detox

siepra commented 4 months ago

Here's a summary of our progress so far:

  1. We set up a physical, self-hosted runner for executing Detox tests for both iOS and Android
  2. We failed to bring multiplayer tests to life, due to simulator incompatibility of Tor binaries
siepra commented 4 months ago

[GitHub hosted Apple Silicon runner]

We succeeded to use GitHub hosted Apple Silicon runner (macos-xlarge) for running iOS Detox tests.

There're problems using similar setup for Android, because of the following problem with running AVD:

HVF error: HV_UNSUPPORTED
qemu-system-aarch64: failed to initialize HVF: Invalid argument

This Stack Overflow thread suggest to re-sign qemu. Unfortunately it doesn't work on CI.

This GitHub issue implies it's a general problem and it's still pretty fresh (the latest comment is only 5 days old for the current day). We believe it's gonna be solved and it's worth to wait.

holmesworcester commented 4 months ago

So we will keep running tests on the local machine in the meantime? That sounds fine to me.

holmesworcester commented 4 months ago

@siepra I was just reading the threads you linked to, and I noticed this comment, which seems to imply that the Android emulator works on the macos-13 runner:

https://github.com/ReactiveCircus/android-emulator-runner/issues/350#issuecomment-2030065889

Have we tried that, including with the workaround you linked to on SO? Could we use macos-13 in the meantime for Android tests?

(Moved out of blocked, but feel free to move it back if we've tried this and it's a dead end! We might as well leave this issue open though.)

holmesworcester commented 4 months ago

Can't do the above due to SDK target

peterlazar1993 commented 1 week ago

I've been doing some research so maybe you'll find this useful. You cannot run a VM(android emulator) inside a VM on M1, M2 macs.

M3 macs with OSX15 (upcoming), will allow nested virtualisation - https://forum.parallels.com/threads/macos-15-sequoia-nested-virtualization-for-m3-macs.364397/