search

thyrlian / AndroidSDK

🐳 Full-fledged Android SDK Docker Image
Apache License 2.0
1.26k stars 218 forks source link
android android-sdk build ci containerization continuous-integration docker docker-container docker-image emulator gradle hack hacking kotlin vnc

readme

AndroidSDK

Android SDK development environment Docker image

Docker Hub Docker Stars Docker Pulls CI Join the chat at https://gitter.im/AndroidSDK-Docker/AndroidSDK-Docker Android Dev Digest AndroidεΌ€ε‘ζŠ€ζœ―ε‘¨ζŠ₯ HelloGitHub Hits

Docker Badge

Conference Talk

Goals

Philosophy

Provide only the barebone SDK (the latest official minimal package) gives you the maximum flexibility in tailoring your own SDK tools for your project. You can maintain an external persistent SDK directory, and mount it to any container. In this way, you don't have to waste time on downloading over and over again, meanwhile, without having any unnecessary package. Additionally, instead of one dedicated Docker image per Android API level (which will end up with a ton of images), you just have to deal with one image. Last but not least, according to Android's terms and conditions, one may not redistribute the SDK or any part of the SDK.

Note

Gradle and Kotlin compiler come together with this Docker image merely for the sake of convenience / trial.

Using the Gradle Wrapper

It is recommended to always execute a build with the Wrapper to ensure a reliable, controlled and standardized execution of the build. Using the Wrapper looks almost exactly like running the build with a Gradle installation. In case the Gradle distribution is not available on the machine, the Wrapper will download it and store in the local file system. Any subsequent build invocation is going to reuse the existing local distribution as long as the distribution URL in the Gradle properties doesn’t change.

Using the Gradle Wrapper lets you build with a precise Gradle version, in order to eliminate any Gradle version problem.

Caveat

Previously, running Android SDK update within the Dockerfile or inside a container would fail with AUFS storage driver, it was due to hardlink move operations (during updating Android SDK) are not supported by AUFS storage driver, but changing it to other storage driver would work. Fortunately, it's not the case any more. With the latest version of Docker Engine, it works like a charm, you can do whatever you prefer. If you're not interested in the technical cause, simply skip this section (jump to the next section).

What happens if the update fails?

  ls $ANDROID_HOME/cmdline-tools/tools/
  #=> empty, nothing is there
  # tools such as: android, sdkmanager, emulator, lint and etc. are gone

  android
  #=> bash: android: command not found

  sdkmanager
  #=> bash: /opt/android-sdk/cmdline-tools/tools/bin/sdkmanager: No such file or directory

To prevent this problem from happening, and you don't wanna bother modifying storage driver. The only solution is to mount an external SDK volume from host to container. Then you are free to try any of below approaches.

If you by accident update SDK on a host machine which has a mismatch target architecture than the container, some binaries won't be executable in container any longer.

  gradle <some_task>
  #=> Error: java.util.concurrent.ExecutionException: java.lang.RuntimeException: AAPT process not ready to receive commands

  $ANDROID_HOME/build-tools/x.x.x/aapt
  #=> aapt: cannot execute binary file: Exec format error

  adb
  #=> adb: cannot execute binary file: Exec format error

Note:

More information about storage driver:

Getting Started

# build the image
# set the working directory to the project's root directory first
docker build -t android-sdk android-sdk
# or you can also pass specific tool version as you wish (optional, while there is default version)
docker build --build-arg JDK_VERSION=<jdk_version> --build-arg GRADLE_VERSION=<gradle_version> --build-arg KOTLIN_VERSION=<kotlin_version> --build-arg ANDROID_SDK_VERSION=<android_sdk_version> -t android-sdk android-sdk
# or pull the image instead of building on your own
docker pull thyrlian/android-sdk

# below commands assume that you've pulled the image

# copy the pre-downloaded SDK to the mounted 'sdk' directory
docker run -it --rm -v $(pwd)/sdk:/sdk thyrlian/android-sdk bash -c 'cp -a $ANDROID_HOME/. /sdk'

# go to the 'sdk' directory on the host, update the SDK
# ONLY IF the host machine is the same target architecture as the container
# JDK required on the host
sdk/cmdline-tools/tools/bin/sdkmanager --update
# or install specific packages
sdk/cmdline-tools/tools/bin/sdkmanager "build-tools;x.y.z" "platforms;android-<api_level>" ...

# mount the updated SDK to container again
# if the host SDK directory is mounted to more than one container
# to avoid multiple containers writing to the SDK directory at the same time
# you should mount the SDK volume in read-only mode
docker run -it -v $(pwd)/sdk:/opt/android-sdk:ro thyrlian/android-sdk /bin/bash

# you can mount without read-only option, only if you need to update SDK inside container
docker run -it -v $(pwd)/sdk:/opt/android-sdk thyrlian/android-sdk /bin/bash

# to keep and reuse Gradle cache
docker run -it -v $(pwd)/sdk:/opt/android-sdk -v $(pwd)/gradle-caches:/root/.gradle/caches thyrlian/android-sdk /bin/bash

# to stop and remove container
# when the image was pulled from a registry
docker stop $(docker ps -aqf "ancestor=thyrlian/android-sdk") &> /dev/null && docker rm $(docker ps -aqf "ancestor=thyrlian/android-sdk") &> /dev/null
# when the image was built locally
docker stop $(docker ps -aqf "ancestor=android-sdk") &> /dev/null && docker rm $(docker ps -aqf "ancestor=android-sdk") &> /dev/null
# more flexible way - doesn't matter where the image comes from
docker stop $(docker ps -a | grep 'android-sdk' | awk '{ print $1 }') &> /dev/null && docker rm $(docker ps -a | grep 'android-sdk' | awk '{ print $1 }') &> /dev/null

Accepting Licenses

A helper script is provided at /opt/license-accepter.sh for accepting the SDK and its various licenses. This is helpful in non-interactive environments such as CI builds.

SSH

It is also possible if you wanna connect to container via SSH. There are three different approaches.

That's it! Now it's up and running, you can ssh to it

  ssh root@<container_ip_address> -p 2222

And, in case you need, you can still attach to the running container (not via ssh) by

  docker exec -it <container_id> /bin/bash

VNC

Remote access to the container's desktop might be helpful if you plan to run emulator inside the container.

  # pull the image with VNC support
  docker pull thyrlian/android-sdk-vnc

  # spin up a container with SSH
  # won't work when spin up with interactive session, since the vncserver won't get launched
  docker run -d -p 5901:5901 -p 2222:22 -v $(pwd)/sdk:/opt/android-sdk thyrlian/android-sdk-vnc

When the container is up and running, use your favorite VNC client to connect to it:

# setup and launch emulator inside the container
# create a new Android Virtual Device
echo "no" | avdmanager create avd -n test -k "system-images;android-25;google_apis;armeabi-v7a"
# launch emulator
emulator -avd test -no-audio -no-boot-anim -accel on -gpu swiftshader_indirect &

For more details, please refer to Emulator section.

VNC client recommendation

NFS

You can host the Android SDK in one host-independent place, and share it across different containers. One solution is using NFS (Network File System).

To make the container consume the NFS, you can try either way below:

And here are instructions for configuring a NFS server (on Ubuntu):

  sudo apt-get update
  sudo apt-get install -y nfs-kernel-server
  sudo mkdir -p /var/nfs/android-sdk

  # put the Android SDK under /var/nfs/android-sdk
  # if you haven't got any, run below commands
  sudo apt-get install -y wget zip
  cd /var/nfs/android-sdk
  sudo wget -q $(wget -q -O- 'https://developer.android.com/sdk' | grep -o "\"https://.*android.*tools.*linux.*\"" | sed "s/\"//g")
  sudo unzip *tools*linux*.zip
  sudo rm *tools*linux*.zip
  sudo mkdir licenses
  echo 8933bad161af4178b1185d1a37fbf41ea5269c55 | sudo tee licenses/android-sdk-license > /dev/null
  echo 84831b9409646a918e30573bab4c9c91346d8abd | sudo tee licenses/android-sdk-preview-license > /dev/null
  echo d975f751698a77b662f1254ddbeed3901e976f5a | sudo tee licenses/intel-android-extra-license > /dev/null

  # configure and launch NFS service
  sudo chown nobody:nogroup /var/nfs
  echo "/var/nfs         *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee --append /etc/exports > /dev/null
  sudo exportfs -a
  sudo service nfs-kernel-server start

Gradle Distributions Mirror Server

There is still room for optimization: recent distribution of Gradle is around 100MB, imagine different containers / build jobs have to perform downloading over and over again, and it has high influence upon your network bandwidth. Setting up a local Gradle distributions mirror server would significantly boost your download speed.

Fortunately, you can easily build such a mirror server docker image on your own.

  docker build -t gradle-server gradle-server
  # by default it downloads the most recent 14 gradle distributions (excluding rc or milestone)
  # or you can also pass how many gradle distributions should be downloaded
  docker build --build-arg GRADLE_DOWNLOAD_AMOUNT=<amount_of_gradle_distributions_to_be_downloaded> -t gradle-server gradle-server

Preferably, you should run the download script locally, and mount the download directory to the container.

  gradle-server/gradle-downloader.sh [DOWNLOAD_DIRECTORY] [DOWNLOAD_AMOUNT]
  docker run -d -p 80:80 -p 443:443 -v [DOWNLOAD_DIRECTORY]:/var/www/gradle.org/public_html/distributions gradle-server
  # copy the SSL certificate from gradle server container to host machine
  docker cp `docker ps -aqf "ancestor=gradle-server"`:/etc/apache2/ssl/apache.crt apache.crt
  # copy the SSL certificate from host machine to AndroidSDK container
  docker cp apache.crt `docker ps -aqf "ancestor=thyrlian/android-sdk"`:/home/apache.crt
  # add self-signed SSL certificate to Java keystore
  docker exec -it `docker ps -aqf "ancestor=thyrlian/android-sdk"` bash -c '$JAVA_HOME/bin/keytool -import -trustcacerts -file /home/apache.crt -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit -noprompt'
  # map gradle services domain to your local IP
  docker exec -it `docker ps -aqf "ancestor=thyrlian/android-sdk"` bash -c 'echo "[YOUR_HOST_IP_ADDRESS_FOR_GRADLE_CONTAINER] services.gradle.org" >> /etc/hosts'

Starting from now on, gradle wrapper will download gradle distributions from your local mirror server, lightning fast! The downloaded distribution will be uncompressed to /root/.gradle/wrapper/dists.

If you don't want to bother with SSL certificate, you can simply change the distributionUrl inside [YOUR_PROJECT]/gradle/wrapper/gradle-wrapper.properties from https to http.

Emulator

ARM emulator is host machine independent, can run anywhere - Linux, macOS, VM and etc. While the performance is a bit poor. On the contrary, x86 emulator requires KVM, which means only runnable on Linux.

According to Google's documentation:

VM acceleration restrictions

Note the following restrictions of VM acceleration:

  • You can't run a VM-accelerated emulator inside another VM, such as a VM hosted by VirtualBox, VMWare, or Docker. You must run the emulator directly on your system hardware.

  • You can't run software that uses another virtualization technology at the same time that you run the accelerated emulator. For example, VirtualBox, VMWare, and Docker currently use a different virtualization technology, so you can't run them at the same time as the accelerated emulator.

Preconditions on the host machine (for x86 emulator)

Read How to Start Intel Hardware-assisted Virtualization (hypervisor) on Linux for more details.

Read KVM Installation if you haven't got KVM installed on the host yet.

Where can I run x86 emulator

How to run emulator

Now you can for instance run UI tests on the emulator (just remember, the performance is POOR):

  <your_android_project>/gradlew connectedAndroidTest

Troubleshooting emulator

If you encounter an error "Process system isn't responding" in the emulator, like below:

You could try:

Access the emulator from outside

Default adb server port: 5037

  # spin up a container
  # with SSH
  docker run -d -p 5037:5037 -p 2222:22 -v $(pwd)/sdk:/opt/android-sdk thyrlian/android-sdk-vnc
  # or with interactive session
  docker run -it -p 5037:5037 -v $(pwd)/sdk:/opt/android-sdk thyrlian/android-sdk-vnc /bin/bash

  # launch emulator inside the container...

Outside the container:

  adb connect <container_ip_address>:5037
  adb devices
  #=> List of devices attached
  #=> emulator-5554 device

Make sure that your adb client talks to the adb server inside the container, instead of the local one on the host machine. This can be achieved by running adb kill-server (to kill the local server if it's already up) before firing adb connect command above.

Android Device

You can give a container access to host's USB Android devices.

  # on Linux
  docker run -it --privileged -v /dev/bus/usb:/dev/bus/usb -v $(pwd)/sdk:/opt/android-sdk thyrlian/android-sdk /bin/bash

  # or
  # try to avoid privileged flag, just add necessary capabilities when possible
  # --device option allows you to run devices inside the container without the --privileged flag
  docker run -it --device=/dev/ttyUSB0 -v $(pwd)/sdk:/opt/android-sdk thyrlian/android-sdk /bin/bash

Note:

Don't worry about adbkey or adbkey.pub under /.android, not required.

Docker for Mac FAQ says:

Unfortunately it is not possible to pass through a USB device (or a serial port) to a container.

Firebase Test Lab

You can also run UI tests on Google's Firebase Test Lab with emulators or physical devices.

To create and configure a project on the platform:

Once finished setup, you can then launch a container to deploy UI tests to Firebase Test Lab:

# pull the image with Google Cloud SDK integrated
docker pull thyrlian/android-sdk-firebase-test-lab

# spin up a container
# don't forget to mount the previously created private key, assume it's saved as firebase.json
# we'll persist all your gcloud configuration which would be created at ~/.config/gcloud/

# spin up a container with interactive mode
docker run -it -v $(pwd)/sdk:/opt/android-sdk -v <your_private_key_dir>/firebase.json:/root/firebase.json -v $(pwd)/gcloud-config:/root/.config/gcloud thyrlian/android-sdk-firebase-test-lab /bin/bash
# or spin up a container with SSH
docker run -d -p 2222:22 -v $(pwd)/sdk:/opt/android-sdk -v <your_private_key_dir>/firebase.json:/root/firebase.json -v $(pwd)/gcloud-config:/root/.config/gcloud -v $(pwd)/authorized_keys:/root/.ssh/authorized_keys thyrlian/android-sdk-firebase-test-lab

# authorize access to Google Cloud Platform with a service account (by its private key)
gcloud auth activate-service-account -q --key-file /root/firebase.json

# list all Android models available for testing
gcloud firebase test android models list

# below are just some examples, to give you an idea how it looks like
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚      MODEL_ID     β”‚        MAKE        β”‚              MODEL_NAME              β”‚   FORM   β”‚  RESOLUTION β”‚      OS_VERSION_IDS     β”‚      TAGS     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Nexus9            β”‚ HTC                β”‚ Nexus 9                              β”‚ VIRTUAL  β”‚ 2048 x 1536 β”‚ 21,22,23,24,25          β”‚               β”‚
β”‚ NexusLowRes       β”‚ Generic            β”‚ Low-resolution MDPI phone            β”‚ VIRTUAL  β”‚  640 x 360  β”‚ 23,24,25,26,27,28,29,30 β”‚ beta=30       β”‚
β”‚ OnePlus3T         β”‚ OnePlus            β”‚ OnePlus 3T                           β”‚ PHYSICAL β”‚ 1920 x 1080 β”‚ 26                      β”‚               β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

# build both the app and the instrumented tests APKs

# build the application APK
./gradlew :<your_module>:assemble
# APK will be generated at: <your_module>/build/outputs/apk/<build_type>

# build the instrumented tests APK
./gradlew :<your_module>:assembleAndroidTest
# APK will be generated at: <your_module>/build/outputs/apk/androidTest/<build_type>

# run UI tests
UI_TEST_APK="--app=<your_apk_path>/debug.apk --test=<your_apk_path>/androidTest.apk"
UI_TEST_TYPE="instrumentation"
UI_TEST_DEVICES="--device model=MODEL_ID,version=OS_VERSION_IDS,locale=en,orientation=portrait"
UI_TEST_RESULT_DIR="build-result"
UI_TEST_PROJECT="<your_project_id>"
gcloud firebase test android run $UI_TEST_APK $UI_TEST_DEVICES --type=$UI_TEST_TYPE --results-dir=$UI_TEST_RESULTS_DIR --project=$UI_TEST_PROJECT

# it's capable of running in parallel on separate devices with a number of shards
# if you want to evenly distribute test cases into a number of shards, specify the flag: --num-uniform-shards=int
# e.g.: to run 20 tests in parallel on 4 devices (5 tests per device): --num-uniform-shards=4

Later you can view the test results (including the recorded video of test execution) in Firebase Console, open your project, navigate to Test Lab.

To learn more about Firebase Test Lab and Google Cloud SDK, please go and visit below links:

Android Commands Reference

Demythologizing Memory

OOM behaviour

Sometimes you may encounter OOM (Out of Memory) issue. The issues vary in logs, while you could find the essence by checking the exit code (echo $?).

For demonstration, below examples try to execute MemoryFiller which can fill memory up quickly.

Facts

Docker Config

How to enable the remote API (for CI purpose)

# on macOS
brew install socat
socat TCP-LISTEN:2376,reuseaddr,fork UNIX-CONNECT:/var/run/docker.sock

#====================================================================================================#

# on Linux (Debian / Ubuntu)
# changing DOCKER_OPTS is optional
# Use DOCKER_OPTS to modify the daemon startup options
# echo -e '\nDOCKER_OPTS="-H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock"\n' | sudo tee --append /etc/default/docker > /dev/null

# find the location of systemd unit file
systemctl status docker
#=> docker.service - Docker Application Container Engine
#=> Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)

sudo sed -i.bak 's?ExecStart.*?ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock?g' /lib/systemd/system/docker.service
sudo systemctl daemon-reload
sudo systemctl restart docker.service

# check if the port is listened or not
sudo netstat -tulpn | grep LISTEN

Release guide

Contributing

Your contribution is always welcome, which will for sure make the Android SDK Docker solution better. Please check the contributing guide to get started! Thank you ☺

Changelog

See here.

Stargazers over time

Stargazers over time

License

Copyright Β© 2016-2024 Jing Li. It is released under the Apache License. See the LICENSE file for details.

By continuing to use this Docker Image, you accept the terms in below license agreement.