search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.93k stars 13.95k forks source link

Package request: Android Open Source Project #21222

Closed spacekitteh closed 5 years ago

spacekitteh commented 7 years ago

Complete overview/guide for downloading AOSP source (including a custom kernel), building and running a QEMU image: https://www.collabora.com/news-and-blog/blog/2016/09/02/building-android-for-qemu-a-step-by-step-guide/

Build environment requirements: https://source.android.com/source/initializing.html Source downloading: https://source.android.com/source/downloading.html Building: https://source.android.com/source/building.html (Included) compiler toolchain: https://source.android.com/source/jack.html Selecting build device targets: https://source.android.com/source/devices.html Kernel-only building: https://source.android.com/source/building-kernels.html Running builds: https://source.android.com/source/running.html

Relevant for nixos (containers/nixops images): QEMU kernel build script: https://android.googlesource.com/platform/prebuilts/qemu-kernel/+/master/build-kernel.sh

moretea commented 7 years ago

See http://sandervanderburg.blogspot.nl/2012/11/building-android-applications-with-nix.html and http://sandervanderburg.blogspot.nl/2014/02/reproducing-android-app-deployments-or.html

Does that answer your question?

spacekitteh commented 7 years ago

No, this is about the build environment for Android itself, rather than Android -apps- :)

On Sun, 18 Dec 2016, 12:43 Maarten Hoogendoorn, notifications@github.com wrote:

See

http://sandervanderburg.blogspot.nl/2012/11/building-android-applications-with-nix.html and

http://sandervanderburg.blogspot.nl/2014/02/reproducing-android-app-deployments-or.html

Does that answer your question?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/21222#issuecomment-267799948, or mute the thread https://github.com/notifications/unsubscribe-auth/AB2crCNRb2RGLuR7tpC3an5TYSSubWVfks5rJJ3cgaJpZM4LPvT_ .

Mic92 commented 7 years ago

Sounds to me like a usecase for https://github.com/NixOS/nixpkgs/pull/21243

moretea commented 7 years ago

@spacekitteh ah, I misread! Apologies. In that case :+1:. I'd like to build a custom build for my Samsung tablet as well ;)

abbradar commented 7 years ago

FWIW, I suceeded building Android on NixOS using an FHS environment:

{ pkgs ? import <nixpkgs> {} }:

let
  jdk = pkgs.jdk8;
  fhs = pkgs.buildFHSUserEnv {
    name = "android-env";
    targetPkgs = pkgs: with pkgs;
      [ git
        gitRepo
        gnupg1
        python2
        curl
        procps
        openssl
        gnumake
        nettools
        androidenv.platformTools
        ccache
        ncurses
        jdk
        schedtool
        utillinux
        m4
        gperf
        perl
        libxml2
        zip
        unzip
        bison
        flex
        lzop
        bc
        which
        gcc
      ];
    multiPkgs = pkgs: with pkgs;
      [ zlib
      ];
    extraOutputsToInstall = [ "dev" ];
    runScript = "bash";
    profile = ''
      export USE_CCACHE=1
      export ANDROID_JAVA_HOME=${jdk.home}
    '';
  };
in fhs.env

Patching all the Android source to get rid of FHS assumptions is a gargantuan and hopeless task IMHO.

spacekitteh commented 7 years ago

Hmm, are you sure that one works? It can't find ncurses, it seems.

Edit: Ahh, it wants ncurses 5. Now it can't find libtinfo... ncurses should create a symlink for libtinfo. Submitted in #21350.

spacekitteh commented 7 years ago

So this is what I currently have:

{ pkgs ? import <nixpkgs> {} }:

let
  jdk = pkgs.jdk8;
  current-repo = "nougat-mr1-release";
  fhs = pkgs.buildFHSUserEnv {
    name = "android-env";
    targetPkgs = pkgs: with pkgs;
      [ git
        gitRepo
        gnupg1compat
        python2
        curl
        procps
        openssl
        gnumake
        nettools
        rsync
        androidenv.buildTools
        androidenv.platformTools
        ccache
        ncurses5
        jdk
        schedtool
        utillinux
        m4
        gperf
        perl
        libxml2
        zip
        unzip
        bison
        flex
        lzop
        bc
        which
        gcc
      ];
    multiPkgs = pkgs: with pkgs;
      [ zlib
      ];
    extraOutputsToInstall = [ "dev" ];
    runScript = "bash";
    profile = ''
      export USE_CCACHE=1
      export ANDROID_JAVA_HOME=${jdk.home}
      export LANG=C
      unset _JAVA_OPTIONS
      export BUILD_NUMBER=$(date --utc +%Y.%m.%d.%H.%M.%S)
      export DISPLAY_BUILD_NUMBER=true
      export USER=copperhead-os-build

      source build/envsetup.sh
    '';
  };
in pkgs.stdenv.mkDerivation {
  name = "android-env-shell";
  nativeBuildInputs = [fhs];
  shellHook = ''
    cd ${current-repo}
    chrt -b -p 0 $$
    exec android-env
  '';
}

(note: requires #21368)

This gets me part of the way there. I run grsecurity, however, so I have to invoke it with sudo; this results in contamination of my /root folder by jack, among other things.

So, I think the next step from here is to containerise it, because AOSP is garbage and I don't want it touching anything on my system.

spacekitteh commented 7 years ago

@abbradar did you ever run into a problem with getting "Unknown cipher suite supported by native code" errors when signing APKs during building?

abbradar commented 7 years ago

@spacekitteh Sadly no. FWIW I built CyanogenMod, not vanilla AOSP -- maybe some differences are in play.

Anyway, do you think we can close this now? I'm not sure Android can/needs to be packaged any further than a chroot environment.

spacekitteh commented 7 years ago

Yeah, I guess.

spacekitteh commented 7 years ago

This needs to be reopened, actually. There are other issues getting it compiling on NixOS, and they probably should be figured out.

The current stumbling block is what appears to be an error with OpenSSL(and LibreSSL, but I'll just say OpenSSL for short) behaving differently on NixOS compared to Debian/Arch/Ubuntu. Specifically, the conscrypt Java Crypto Provider doesn't seem to be able to work properly with OpenSSL as nixpkgs provides. It gives the following error:

[ 25% 10601/42050] target Package: BasicDreams (out/targe...ct/marlin/obj/APPS/BasicDreams_intermediates/package.apk)

FAILED: /bin/bash -c "(touch out/target/product/marlin/obj/APPS/BasicDreams_intermediates/zipdummy ) && ((cd out/target/product/marlin/obj/APPS/BasicDreams_intermediates/ && jar cf package.apk zipdummy) ) && (zip -qd out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk zipdummy ) && (rm out/target/product/marlin/obj/APPS/BasicDreams_intermediates/zipdummy ) && (out/host/linux-x86/bin/aapt package -u -z  --pseudo-localize -c en_US,en_US,cs_CZ,da_DK,de_AT,de_CH,de_DE,de_LI,el_GR,en_AU,en_CA,en_GB,en_NZ,en_SG,eo_EU,es_ES,fr_CA,fr_CH,fr_BE,fr_FR,it_CH,it_IT,ja_JP,ko_KR,nb_NO,nl_BE,nl_NL,pl_PL,pt_PT,ru_RU,sv_SE,tr_TR,zh_CN,zh_HK,zh_TW,am_ET,hi_IN,en_US,en_AU,en_IN,fr_FR,it_IT,es_ES,et_EE,de_DE,nl_NL,cs_CZ,pl_PL,ja_JP,zh_TW,zh_CN,zh_HK,ru_RU,ko_KR,nb_NO,es_US,da_DK,el_GR,tr_TR,pt_PT,pt_BR,sv_SE,bg_BG,ca_ES,en_GB,fi_FI,hi_IN,hr_HR,hu_HU,in_ID,iw_IL,lt_LT,lv_LV,ro_RO,sk_SK,sl_SI,sr_RS,uk_UA,vi_VN,tl_PH,ar_EG,fa_IR,th_TH,sw_TZ,ms_MY,af_ZA,zu_ZA,am_ET,en_XA,ar_XB,fr_CA,km_KH,lo_LA,ne_NP,si_LK,mn_MN,hy_AM,az_AZ,ka_GE,my_MM,mr_IN,ml_IN,is_IS,mk_MK,ky_KG,eu_ES,gl_ES,bn_BD,ta_IN,kn_IN,te_IN,uz_UZ,ur_PK,kk_KZ,sq_AL,gu_IN,pa_IN,be_BY,bs_BA,normal,xlarge,large --preferred-density 560dpi -M packages/screensavers/Basic/AndroidManifest.xml -S packages/screensavers/Basic/res  -I out/target/common/obj/APPS/framework-res_intermediates/package-export.apk --min-sdk-version 25 --target-sdk-version 25 --product nosdcard --version-code 25 --version-name 7.1.1   --skip-symbols-without-default-localization -F out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk ) && (find out/target/common/obj/APPS/BasicDreams_intermediates/ -maxdepth 1 -name \"classes*.dex\" | sort | xargs zip -qjX out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk ) && (if [ -d out/target/common/obj/APPS/BasicDreams_intermediates/jack-rsc ] ; then find out/target/common/obj/APPS/BasicDreams_intermediates/jack-rsc -type f | sort | sed -e \"s?^out/target/common/obj/APPS/BasicDreams_intermediates/jack-rsc/? -C \\\"out/target/common/obj/APPS/BasicDreams_intermediates/jack-rsc\\\" \\\"?\" -e\"s/\$/\\\"/\" > out/target/product/marlin/obj/APPS/BasicDreams_intermediates/jack_res_jar_flags; if [ -s out/target/product/marlin/obj/APPS/BasicDreams_intermediates/jack_res_jar_flags ] ; then jar uf out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk @out/target/product/marlin/obj/APPS/BasicDreams_intermediates/jack_res_jar_flags; fi; fi ) && (mv out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk.unsigned ) && (java -Djava.library.path=out/host/linux-x86/lib64 -jar out/host/linux-x86/framework/signapk.jar --min-sdk-version \$((out/host/linux-x86/bin/aapt dump badging out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk.unsigned 2>&1 | grep '^sdkVersion' || echo \"sdkVersion:'0'\") | cut -d\"'\" -f2 | sed -e s/^.*[^0-9].*\$/25/) build/target/product/security/testkey.x509.pem build/target/product/security/testkey.pk8  out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk.unsigned out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk.signed ) && (mv out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk.signed out/target/product/marlin/obj/APPS/BasicDreams_intermediates/package.apk )"

Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.conscrypt.OpenSSLBIOInputStream.<init>(OpenSSLBIOInputStream.java:34)
        at org.conscrypt.OpenSSLX509Certificate.fromX509PemInputStream(OpenSSLX509Certificate.java:119)
        at org.conscrypt.OpenSSLX509CertificateFactory$1.fromX509PemInputStream(OpenSSLX509CertificateFactory.java:220)
        at org.conscrypt.OpenSSLX509CertificateFactory$1.fromX509PemInputStream(OpenSSLX509CertificateFactory.java:216)
        at org.conscrypt.OpenSSLX509CertificateFactory$Parser.generateItem(OpenSSLX509CertificateFactory.java:94)
        at org.conscrypt.OpenSSLX509CertificateFactory.engineGenerateCertificate(OpenSSLX509CertificateFactory.java:272)
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
        at com.android.signapk.SignApk.readPublicKey(SignApk.java:182)
        at com.android.signapk.SignApk.main(SignApk.java:1087)
Caused by: java.lang.IllegalArgumentException: Unknown cipher suite supported by native code: DH-DSS-AES256-GCM-SHA384
        at org.conscrypt.NativeCrypto.<clinit>(NativeCrypto.java:750)
        ... 9 more
[ 25% 10601/42050] target thumb C++: libLLVMSelectionDAG_...<= external/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp
ninja: build stopped: subcommand failed.
make: *** [build/core/ninja.mk:149: ninja_wrapper] Error 1

At first I thought it was a bug in the way OpenSSL filters worked which NixOS exposes (see #21408) but apparently not.

Issue upstream in conscrypt is here: https://github.com/google/conscrypt/issues/20

evrim commented 7 years ago

ping @spacekitteh @abbradar 

{ pkgs ? import  {} }:

let
  jdk =  pkgs.callPackage <nixpkgs/pkgs/development/compilers/openjdk/8.nix> {
    bootjdk = pkgs.callPackage <nixpkgs/pkgs/development/compilers/openjdk/bootstrap.nix> { version = "8"; };
    inherit (pkgs.gnome2) GConf gnome_vfs;
    minimal = true;
  };
  fhs = pkgs.buildFHSUserEnv {
    name = "android-env";
    targetPkgs = pkgs: with pkgs;
      [ git
        gitRepo
        gnupg1compat
        python2
        curl
        procps
    openssl
    gnumake
    nettools
    rsync
    androidenv.buildTools
    androidenv.platformTools
    ccache
    ncurses5
    jdk
    schedtool
    utillinux
    m4
        gperf
    perl
    libxml2
    zip
    unzip
    bison
    flex
    lzop
    bc
        which
    gcc
    binutils
    coreutils
    gdb
      ];
    multiPkgs = pkgs: with pkgs;
      [ zlib
      ];
    extraOutputsToInstall = [ "dev" ];
    runScript = "bash";
    profile = ''
      export USE_CCACHE=1
      export ANDROID_JAVA_HOME=${jdk.home}
      export LANG=C
      unset _JAVA_OPTIONS
      export BUILD_NUMBER=$(date --utc +%Y.%m.%d.%H.%M.%S)
      export DISPLAY_BUILD_NUMBER=true
    '';
  };
in fhs.env

bash-4.4$ ldd $(which java)
    linux-vdso.so.1 (0x000075585ff92000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x000075585fb53000)
    libz.so.1 => /usr/lib/libz.so.1 (0x000075585f93c000)
    libjli.so => /nix/store/xbr1i9c1gmz4if9l8g75pw6xzj7s1d9n-openjdk-8u141b15-jre/lib/openjdk/jre/lib/amd64/jli/libjli.so (0x000075585f72c000)
    libdl.so.2 => /usr/lib/libdl.so.2 (0x000075585f528000)
    libc.so.6 => /usr/lib/libc.so.6 (0x000075585f189000)
    /nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/ld-linux-x86-64.so.2 (0x000075585fd71000)
evrim commented 7 years ago 
[aosp/out/repo/host/linux-x86/bin]# paxctl -vQ ./dex2oatd
PaX control v0.9
Copyright 2004,2005,2006,2007,2009,2010,2011,2012,2014 PaX Team 

- PaX flags: -p-s-m-x-e-r [./dex2oatd]
evrim commented 7 years ago

For 6.0.x

{ pkgs ? import  {} }:

let
  jdk =  pkgs.callPackage  {
    bootjdk = pkgs.callPackage  { version = "7"; };
    minimal = true;
  };
  fhs = pkgs.buildFHSUserEnv {
    name = "android-env";
    targetPkgs = pkgs: with pkgs;
      [ git gitRepo gnupg1compat python2 curl procps openssl gnumake nettools
        rsync androidenv.buildTools androidenv.platformTools ccache ncurses5 jdk schedtool utillinux
        m4 gperf perl libxml2 zip unzip bison flex lzop bc which gcc binutils coreutils gdb python
        emacs vim
      ];
    multiPkgs = pkgs: with pkgs;
      [ zlib
      ];
    extraOutputsToInstall = [ "dev" ];
    runScript = "bash";
    profile = ''
      export USE_CCACHE=1
      export ANDROID_JAVA_HOME=${jdk.home}
      export LANG=C
      unset _JAVA_OPTIONS
      export BUILD_NUMBER=$(date --utc +%Y.%m.%d.%H.%M.%S)
      export DISPLAY_BUILD_NUMBER=true
    '';
  };
in fhs.env
Ekleog commented 6 years ago

(triage) Is this still a problem? I know someone who recently built an Android OS using the instructions at https://nixos.wiki/wiki/Android.

ajs124 commented 5 years ago

This is definitely not a problem anymore. I have successfully built LineageOS 15.1 and 16.0, so Android 8.1 and 9.0 on NixOS.

Ekleog commented 5 years ago

Closing for now, then.