search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.07k stars 14.05k forks source link

Build Android tools from source (adb, SDK, Android Studio, etc.) #75603

Open primeos opened 4 years ago

primeos commented 4 years ago

We should build as much Android tools as possible from source. The most relevant tool is probably the adb (Android Debug Bridge) to communicate with Android phones.

Reason

The official builds from Google are distributed under an unfree license (see #53403).

Difficulty

Probably high.

Resources

matthewbauer commented 4 years ago

Related to:

Atemu commented 4 years ago

One more reason to do this: Nixpkgs' ADB is only available on x86

tgunnoe commented 4 years ago

It is currently impossible to accept the android license when the sdk sits in the nix store, and is installed through nixpkgs. So this is probably needed.

https://github.com/NixOS/nixpkgs/pull/76420#issuecomment-615344955

danielfullmer commented 4 years ago

First, note there is an android rebuilds project which aims to provide license unencumbered builds of the android SDK.

Also, I've been working on a project (robotnix) to build Android ROMs using nix.

As an experiment, I looked into whether it could be leveraged to build the android SDK, with some success: https://github.com/danielfullmer/robotnix/blob/master/sdk/default.nix To build:

nix-build -E 'import ((builtins.fetchTarball "https://github.com/danielfullmer/robotnix/archive/master.tar.gz") + "/sdk")'

Which produces the following files under ./result:

android-sdk_eng.10.0.0_r33_linux-x86.zip
repository.xml
repo-sys-img.xml
sdk-repo-linux-build-tools-eng.10.0.0_r33.zip
sdk-repo-linux-docs-eng.10.0.0_r33.zip
sdk-repo-linux-platforms-eng.10.0.0_r33.zip
sdk-repo-linux-platform-tools-eng.10.0.0_r33.zip
sdk-repo-linux-samples-eng.10.0.0_r33.zip
sdk-repo-linux-sources-eng.10.0.0_r33.zip
sdk-repo-linux-system-images-eng.10.0.0_r33.zip

There are a few caveats: 1) The source is huge. >40GB is required just to check out the AOSP source. To avoid extraneous copying, I use namespaces to bind-mount the source into its proper location during the build. 2) It takes a very long time to build. It takes approximately 4 hours on my i3770. I have since upgraded to a 32-core threadripper which can do this in about 25 minutes. 3) It relies on Google's prebuilt toolchain included in the AOSP checkout. This only has support for linux-x86 and darwin-x86. We still have to autoPatchelf the tools which are intended to be run on the host. 4) It relies on the module system I'm using for robotnix, which probably doesn't belong in nixpkgs. This could (with a little work) be removed to produce some relatively simple nix code for building the sdk.

Instead of building the entire SDK, I also looked into building only adb and fastboot here: https://github.com/danielfullmer/robotnix/blob/master/sdk/adb.nix This is much faster, but unfortunately still requires checking out the entire AOSP source. I spent a little time trying to exclude various subprojects, but soong (android's new build system) would complain about missing references.

As far as inclusion in nixpkgs: I think an alternative package for adb/fastboot which doesn't require accepting the SDK license would be most valuable for users. However, I'm not sure such a package would be accepted into nixpkgs if it requires checking out all of AOSP.

xaverdh commented 4 years ago

Instead of building the entire SDK, I also looked into building only adb and fastboot here: https://github.com/danielfullmer/robotnix/blob/master/sdk/adb.nix This is much faster, but unfortunately still requires checking out the entire AOSP source. I spent a little time trying to exclude various subprojects, but soong (android's new build system) would complain about missing references.

As far as inclusion in nixpkgs: I think an alternative package for adb/fastboot which doesn't require accepting the SDK license would be most valuable for users. However, I'm not sure such a package would be accepted into nixpkgs if it requires checking out all of AOSP.

Maybe somebody with more knowledge of the hydra build servers / infra can tell if that would be a problem.

But is soong actually insisting on unneeded references, or are all of those really needed? Maybe you could trace the build process and just track all files read during building, then set up a git repository with just these files in it? If it works well, this could be a fully automated process.

danielfullmer commented 3 years ago

@xaverdh So the cause of the soong failure is that it walks the entire android source tree and tries to resolve all the build references in the blueprint files before building anything. However, I recently found that soong has an ALLOW_MISSING_DEPENDENCIES environment variable, which does seem to ignore some missing dependencies. I've tried using this environment variable and adding only the repos based on the error messages I get while building adb/fastboot. Even with this, I still get a number of depends on undefined module errors that refer to modules that seem entirely unrelated to adb/fastboot. It might be possible to resolve these issues with additional work...

musfay commented 3 years ago

If we only want to build tools, there is a friendly build-system for it. https://github.com/nmeum/android-tools Fedora and Arch already switched to it, openSUSE devs are also thinking about switching to it. The source tarball is about 24mb. Also we can create our tarballs with build-system. I think it's worth a look.

primeos commented 3 years ago

@musfay that's really awesome, thanks a lot for mentioning it! Seems like the Alpine Linux package maintainer created this and Void Linux is also using and contributing to it. I've drafted #124992 for Nixpkgs but I/we need to figure out how to best fetch the vendored Go modules first (then the rest of the build will hopefully succeed).

samueldr commented 3 years ago

124992 is now merged, closing :)

malte-v commented 3 years ago

This should be reopened. Currently we're building adb and fastboot from source, but the SDK and Android Studio are still proprietary binaries with license agreements attached to them.

samueldr commented 3 years ago

Indeed. Another issue with a broad scope which becomes hard to close.

malte-v commented 3 years ago

Probably relevant: https://github.com/danielfullmer/robotnix/issues/107

stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info

Ericson2314 commented 2 years ago

Keep alive, we got closer with Android cross, and perhaps robotnix can help.

hello-smile6 commented 2 years ago

Is it likely that this will work soon?

xaverdh commented 2 years ago

Well the nixos adb module could already use adb built from source (I opened https://github.com/NixOS/nixpkgs/pull/136040 for that). Everything else will probably take some time.

zkpgbig commented 1 year ago

If we only want to build tools, there is a friendly build-system for it. https://github.com/nmeum/android-tools Fedora and Arch already switched to it, openSUSE devs are also thinking about switching to it. The source tarball is about 24mb. Also we can create our tarballs with build-system. I think it's worth a look. I want to compile build-tools located in build/tools folder in source. make sdk-eng?

Atemu commented 1 year ago

@zkpgbig android-tools is already packaged from source.

zkpgbig commented 1 year ago

@zkpgbig android-tools is already packaged from source.

I just want to build it from source…

Atemu commented 1 year ago

nix-build '<nixpkgs>' -A android-tools