search

mmcc007 / test_emulators

Test start-up of android emulators on Travis and Cirrus. Plus run a test.
14 stars 7 forks source link

Support for android x86 and x86_64 system images in latest emulator canary #2

Open msand opened 5 years ago

msand commented 5 years ago

Thanks for you nice test repo! I managed to get x86 to start on travis just now: https://travis-ci.org/msand/test_emulators/jobs/502382370 Seems quite a few of them are working: https://travis-ci.org/msand/test_emulators/builds/502382369

mmcc007 commented 5 years ago

Thanks for the kind words.

Those x86 emulators start really fast! Is there something new about these emulators? It's definitely worth more investigation.

I think these emulators are starting because of the xvfb (X-Windows Virtual Framebuffer). I tried running a test on several emulators running on xvfb and the x86 emulator and the test would not run. I then tried without xvfb on a known good ARM emulator and it runs (as expected)... without xvfb the x86 emulators won't start.

You can see my attempts at: https://travis-ci.org/mmcc007/test_emulators/builds on the dev branch

I added an actual android test (based on Flutter) that I am using to consider a emulator usable. I will probably update the list to show good and bad emulators by running an android test.

If you can find one of these fast-starting emulators that will run an actual android test then that's a major win!... let me know if you find one!

For now, I will probably remove the xvfb as, so far, no android test actually runs with it.

Anyway, lots of test configurations to try out...

msand commented 5 years ago

The main change is to use the new canary android emulator, it has headless support and the -no-accel flag: https://androidstudio.googleblog.com/2019/02/emulator-2818-canary.html I'm using it in here: https://github.com/msand/react-native-svg-e2e/commits/master Running on travis here: https://travis-ci.org/msand/react-native-svg-e2e/builds

But I'm using "system-images;android-21;google_apis;armeabi-v7a" with the normal emulator rather than the headless, was encountering some issues with x86 on travis.

On bitrise I have it using "system-images;android-28;google_apis;x86" with the headless version, running on a mac, didn't get it running e2e tests on linux there yet, stalls when installing the apk. Here's the latest successful run: https://app.bitrise.io/build/f9c63b00ee41121f

msand commented 5 years ago

Links to latest canary versions can be found here: https://aur.archlinux.org/packages/android-emulator-canary/

mikehardy commented 4 years ago

It appears Travis just did some kind of infrastructure refresh and I haven't tested it yet but maybe x86 nested virtualization works now? The x86 emulators with major version 29 (the ones previously mentioned here as being on the canary channel) have been working great for me for months now without acceleration, at least

mmcc007 commented 4 years ago

Something seems to be changing in the world of CI vendors. Cirrus-CI contacted me to see if they could experiment with a sample app on their infrastructure with hardware acceleration (using a KVM container). https://github.com/fkorotkov/screenshots/tree/fedor/cirrus-ci (couldn't find the actual build https://cirrus-ci.com/github/fkorotkov but apparently they are planning on making an announcement)

Might be worth experimenting with the cirrus kvm container on a simple android app with a test (assuming it is publicly available).

(would also be great to see a simple android example app running on an x86 emulator to use as a reference. The examples provided by @msand are great but a little hard to follow for me -- I'm not a react-native expert)

(@mikehardy do you have a link to a travis config file for the x86 emulators you are using?)

mikehardy commented 4 years ago

Travis will do KVM now apparently - here's a sketch of a workplan: https://github.com/ankidroid/Anki-Android/issues/5583

My .travis.yml - https://github.com/ankidroid/Anki-Android/blob/master/.travis.yml

I think it's just that it is really really hard to make changes in these CI systems so they don't move fast. And the cloud vendors themselves are just getting it enabled. But now it appears that yes, it's making it's way out everywhere.

mikehardy commented 4 years ago

It all works now - android x86/x86_64 emulators up to the current ones (API16-29, not including 26 only), with hardware acceleration on travis supported by nested virtualization. https://github.com/ankidroid/Anki-Android/pull/5594 / https://travis-ci.org/ankidroid/Anki-Android/builds/606100691

mmcc007 commented 4 years ago

That's great news!

Also adapted for this project. Had to disable caching for now to get consistent results (gradlew crashes with "Could not load compiled classes for settings file"). Added Flutter test that passed for all APIs except 16 and 17.

For shits and giggles, also tried to run on mac (even though there is not much point). The mac platform requires Intel's HAXM (or Apple's Hypervisor.Framework) to enable x86 acceleration for android emulator. This is not enabled yet on the mac (I assume Travis has to enable acceleration in the CPU and install the kernel drivers for this to work, ie, can't enable via .travis.yml). So bypassed builds on mac for now (the identical flutter test can run reliably on iOS simulator without acceleration anyway, but running flutter tests on iOS simulator is out-of-scope for this project). When x86 acceleration is eventually enabled on mac, android emulator should run on mac too with minor changes to .travis.yml.

To see an android emulator running a flutter test on the same APIs as used by @Overflow0xFFFF: https://travis-ci.org/mmcc007/test_emulators/builds/606291855

mmcc007 commented 4 years ago

Also got it working on Cirrus-CI (simpler to configure and runs faster) https://cirrus-ci.com/build/6037328228777984 https://github.com/mmcc007/test_emulators/blob/dd18714ad6f02115c8d76c3fbb53038024ca1203/.cirrus.yml fails: 25 cirrus, travis 18 out of 70

mikehardy commented 4 years ago

Interesting. Cirrus is new for me but they are certainly offering more CPU grunt, it's faster because they are dishing out 4 CPUs and 10GB RAM to OSS projects, Travis works with 2CPU and less RAM. Otherwise seems roughly equivalent except their android environment works out of the box :-) (for now...)

mmcc007 commented 4 years ago

On Cirrus, can also specify your own docker images that it stores locally and spin-up very fast. (see .cirrus.yml and https://cirrus-ci.com/build/4728119050633216)

Max on Cirrus is 8.0 CPUs and 24 Gb for open source, but apparently the larger configs take longer to start.

Also for parallel runners, you get 8 linux + 1 macOS on Cirrus for a total of 9 v's 5 (or 3) linux + 2 macOS on Travis for a max of 5 (or 3) parallel runners.

Plus cirrus runner limits are per person, travis limits are per project. So on cirrus, if 3 people working on the same project you'll be able to run total of 24 Linux and 3 macOS runners in parallel.

AFAIK both are on Google Cloud.