search

nix-community / robotnix

Build Android (AOSP) using Nix [maintainer=@danielfullmer,@Atemu]
616 stars 69 forks source link

Add documentation on how to get started (currently struggling with this) #94

Closed codethief closed 3 years ago

codethief commented 3 years ago

Hi Daniel, thank you so much for this really exciting project! I've been looking for something like this for a long time!

So I've been trying to follow the README to build GrapheneOS for my Pixel 5 (which seems like it's already supported(?) but I'm not entirely sure). In any case, I tried building GrapheneOS for crosshatch just to figure out how things work but I gotta admit that I'm completely new to Nix (and relatively new to building Android), so I've been struggling a bit.

I would therefore suggest elaborating a bit on the "Configuration and Build Options" part of the README and adding a file like docs/getting-started.md that provides a little more hand-holding to beginners like me. Maybe one could even show how to set everything up inside a Docker container (which is what I'm planning to do). I'm saying "one" here, not "you", because I would offer writing this myself, once I've got everything working. :) What do you think?

In the meantime, while I'm trying to get things to work, I hope it's ok if I exploit this thread and ask a couple beginner's questions. Here are the steps that I followed:

$ # (Install Nix)
$ git clone https://github.com/danielfullmer/robotnix.git
$ cd robotnix
$ # (Now I edited example.nix and uncommented the 'device' and 'flavor' settings)
$ nix-build . --arg configuration ./example.nix -A img
error: The option `signing.avb.fingerprint' is used but not defined.

At this point I tried playing around with the signing.avb.* settings I found on https://docs.robotnix.org/options.html [0] but no matter what I do (short of setting signing.avb.fingerprint to a 64-character fingerprint string, like here in your personal config), I always get an error, despite the fact that AVB signing should be disabled by default. (Btw, what exactly is the difference between signing.enable and signing.avb.enable?)

Interestingly, 

$ nix-build "https://github.com/danielfullmer/robotnix/archive/master.tar.gz" --arg configuration '{ device="crosshatch"; flavor="grapheneos"; }' -A img

works just fine. So I'm wondering what's so different about the example.nix file.

[0]: I actually don't remember anymore where I found https://docs.robotnix.org/options.html but in any case that site deserves prominent placement in the README. :)

hmenke commented 3 years ago

You have signing disabled but the Auditor app is enabled, which requires signing. This works for me:

--- a/example.nix
+++ b/example.nix
@@ -11,8 +11,8 @@ let
 in
 {
   # These are required options, but commented out here since I set it programmatically for my devices elsewhere
-  #device = "crosshatch";
-  #flavor = "grapheneos"; # "vanilla" is another option
+  device = "bramble";
+  flavor = "grapheneos"; # "vanilla" is another option

   # buildDateTime is set by default by the flavor, and is updated when those flavors have new releases.
   # If you make new changes to your build that you want to be pushed by the OTA updater, you should set this yourself.
@@ -33,8 +33,8 @@ in
     fdroid.enable = true;

     # See the NixOS module under nixos/ subdir
-    auditor.enable = true;
-    auditor.domain = "attestation.${myDomain}";
+    #auditor.enable = true;
+    #auditor.domain = "attestation.${myDomain}";
   };

   # Custom hosts file
hmenke commented 3 years ago

Docs are not yet merged, see https://github.com/danielfullmer/robotnix/pull/88.

hmenke commented 3 years ago

@danielfullmer A quick and dirty fix for this problem would be

--- a/modules/apps/auditor.nix
+++ b/modules/apps/auditor.nix
@@ -27,7 +27,7 @@ in
       message = "Device ${config.device} is currently unsupported for use with auditor app.";
     } ];

-    apps.prebuilt.Auditor = {
+    apps.prebuilt.Auditor = assert assertMsg config.signing.enable "Auditor App requires signing"; {
       apk = apks.auditor.override {
         inherit (cfg) domain;
         inherit (config) device;

A module-level assertion does not work here, because the fingerprint are pulled in via IFD which is evaluated before merging the attrsets and therefore evaluates before any module assertion.

codethief commented 3 years ago

Thank you so much, @hmenke ! I'll test this in the coming days and will report back!

codethief commented 3 years ago

Hi @hmenke ! Thank you again for your help – it seems to be working now!

I've got a few follow-up questions:

  1. What is the relation between the nix-build -A img and -A releaseScript build targets? I just attempted to create the release script (just to test things) and it ended up pulling dozens of gigabytes of dependencies until I ran out of HDD space. Does the release script also build the image if it doesn't exit yet or are building and releasing two separate steps? The reason I'm asking is that ./release ./keys does not include a path to the image. My plan was to build the image on a cloud server, download it to my local machine, create the release script on my local machine (or use nix-copy-closure) and then do the release but, given that the script needs so much space, I'm not sure that's feasible.
  2. The docs say that signing.avb.fingerprint is the hash of avb_pkmd.bin. What is this file and where I do I get it from?
  3. I just generated the generate-keys script and executed it and was surprised to find out that it also generates keys for F-Droid and MicroG, among others. From this I conclude that F-Droid and MicroG also get built by the build script – correct?
hmenke commented 3 years ago

I'm not so deep in the material, so take everything I say with a grain of salt.

  1. If you are not signing your build there is no distinction, but if signing is enabled, the img will be signed during the Nix build, whereas releaseScript will build unsigned target files and a script that you can then run outside of Nix. You should build the releaseScript on the cloud server and then copy its closure to your local machine. It has all the necessary paths baked in and only needs a path to the keys.

  2. When you ran the generateKeysScript, it created a file ./keys/<device>/avb_pkmd.bin. To get the hash you can use

    sha256sum keys/<device>/avb_pkmd.bin | awk '{print toupper($1)}'

  3. Short answer: Android requires all APKs to be signed. Long answer: I think robotnix is using prebuilt APKs for F-Droid and microG, which should already be signed by the respective parties, but for example the Auditor app is built from source so it definitely needs it's own signing key. So F-Droid and microG are being resigned with own keys with eyes on a future where they are built from source as well. Probably you could sign all APKs with the same platform key as well, but that would make it impossible to revoke individual signing keys if you need to.

Right now the whole signing business is a bit wonky, so personally I'm actually signing the build product during the Nix build with a sandbox exception, but I'm also running the build on a local machine. If it helps you, here are are robotnix configurations and build scripts: https://git.henrimenke.de/henri/android

codethief commented 3 years ago

Thank you for your detailed response, @hmenke !

When you ran the generateKeysScript, it created a file ./keys/<device>/avb_pkmd.bin

Ah, I see. I'm surprised this file's fingerprint needs fixating in the config, though, whereas all other keys in ./keys/<device> just get pulled from the file.

codethief commented 3 years ago

Just an FYI for other people running into the same issues:

patching script interpreter paths in ./infra/config/dev.star
patching script interpreter paths in ./infra/scripts/sizes.py
./infra/scripts/sizes.py: interpreter directive changed from "/usr/bin/env python" to "/nix/store/s3rb12xyvlba5cgzh9sf428bxsv3b3kp-python-2.7.18-env/bin/python"
configuring
Done. Made 41714 targets from 2565 files in 20518ms
building
** Message: 00:12:16.631: Requires Linux version >= 3.19 built with CONFIG_USER_NS
** Message: 00:12:16.639: Run: sudo sysctl -w kernel.unprivileged_userns_clone=1

** (process:440700): ERROR **: 00:12:16.639: main: unshare: Operation not permitted
/nix/store/av0pavd6vn698g82ml66gd2hnjd01nzb-stdenv-linux/setup: line 1305: 440698 Trace/breakpoint trap   chromium-fhs  <<'EOF'
set -euo pipefail
cd src
ninja -C out/Release chrome_modern_public_bundle system_webview_apk | cat
EOF

builder for '/nix/store/h68qm9qg1nprs8x1rcba87nngw3av4bf-vanadium-89.0.4389.105.drv' failed with exit code 133

I have yet to figure out how to fix this (on my machine cat /proc/sys/kernel/unprivileged_userns_clone already returns 1) – it seems to be related to Docker. Fortunately, though, I don't need Vanadium in the first place, so I'll just disable Vanadium for now (see above).

codethief commented 3 years ago

Turns out, even after explicitly setting app.vanadium.enable = false; in my config, Vanadium still gets built.

hmenke commented 3 years ago
codethief commented 3 years ago

Vanadium will still be built since it is required as the system webview

Huh, that's at least surprising. The logs say that Nix is trying to build a derivation /nix/store/aq5r25ap0hzb9l25294fqk0bn0pjh4s2-prebuilt_vanadiumwebview.drv, so from that I would have assumed that the WebView dependency does not trigger building of Vanadium.

You need Nix with sandboxing enabled for robotnix to work.

Thanks for the heads-up! So I guess I'll have to avoid Docker for the time being.

hmenke commented 3 years ago

Ah, the name prebuilt is maybe a bit misleading here. It just means that this is not built as part of the Android build process.

In case you haven't noticed, there is also a binary cache for kernels and webviews: https://github.com/danielfullmer/robotnix#binary-cache

codethief commented 3 years ago

You shouldn't have to set signing.avb.fingerprint manually if the keys are available during build time.

I'm still struggling with the whole AVB business: So I hadn't set signing.enable or signing.avb.enable (let alone signing.avb.fingerprint) at all in my config. Then, using that config, I built the -A releaseScript target and later on ran the release script with my keys. In its logs I noticed that it replaced all the test keys from the original build with my own keys, as expected.

Then I noticed that the resulting $DEVICE-factory-YYYY-MM-DD.zip did not include the avb_pkmd.bin file (like in the official GrapheneOS builds) and the flash-all.sh script also didn't include the corresponding flashboot commands to set the keys. So far, so good, I thought – I know that I didn't provide the fingerprint in my config as you had told me to.

So I added the following lines to my config:

signing.avb.enable = true;
signing.avb.fingerprint = "the fingerprint";

and reran nix-build -A releaseScript, expecting the release script to change to account for my changes in some way. Nothing happened, though.

Looking at the code, it seems signing.avb.fingerprint is only relevant for the auditor – which I hadn't enabled in my build config to begin with. Moreover, signing.avb.enable is set to true by default for Pixel devices like mine, anyway, – another instance of the docs betraying me –, so it makes sense that the release script didn't change at all because my config effectively hadn't changed.

What I still don't understand, though, is why my factory.zip doesn't include the avb_pkmd.bin? Is this something that only GrapheneOS does and I have to flash avb_pkmd.bin on my own?

More generally, I still don't comprehend the role of the signing.enable option. If I want signed builds at the end of the day, do I only have to enable it when building -A img or also for -A releaseScript? My impression from these lines had been that if I activate signing.enable, I will also have to provide the keys in /keys, and only if I leave it disabled, it will use test keys (source 1, source 2) during the build process until the release script later replaces them with the production keys. Is this correct? What am I missing?

Sorry if I'm asking questions here whose answers would be obvious to a Nix veteran but I'm really struggling to understand the Nix code.

danielfullmer commented 3 years ago

Apologies for the delay responding to your comments, and thanks for your persistence. You're running into some rough edges around fingerprints/signing that I realize are not the best experience. (https://github.com/danielfullmer/robotnix/issues/24).

Yes, you'll currently need to flash avb_pkmd.bin yourself (See https://docs.robotnix.org/installation.html). I've created https://github.com/danielfullmer/robotnix/issues/98 to keep track of the issue. That functionality in GrapheneOS was added some time after I'd created the GrapheneOS flavor and I didn't notice it until recently.

You set signing.enable = true; if you want to have signed builds, no matter whether you use -A img or -A releaseScript. Enabling signing.enable will sometimes mean that certain fingerprints need to be included deep inside the build (e.g. F-Droid, Auditor, etc.), so it can sometimes trigger an entire rebuild. The signing.avb.fingerprint and apps.prebuilt.<name>.fingerprint options should be filled in by default based on the key directory specified in keyStorePath, using a somewhat frowned upon feature from Nix called "import-from-derivation". Later, I'm hoping to avoid that by having the generateKeysScript produce the required keys fingerprints/metadata as a separate configuration file that can be included in your build.

I added a note to the options reference for signing.avb.fingerprint and apps.prebuilt.<name>.fingerprint saying that they should normally be set automatically. I also added a note to the options reference page pointing out that various flavors / modules could potentially change the default settings of other options (as you've noticed with apps.vanadium.enable for GrapheneOS).

Referring to an earlier question you had: the releaseScript does internally refer to a Nix-built target-files, which it uses in conjunction with your keys to produce the final signed products. If you want to do your build on an instance in the cloud, you should create the releaseScript on the cloud image (which will build most of the android stuff), and then use nix-copy-closure on the resulting releaseScript to copy it down to your local machine.

Finally, the F-Droid app is built from source in robotnix, while MicroG is not (I had difficulties getting it to build inside Nix). Most apps like this should get their own keys instead of using the device-specific platform keys. In the future, we might share the exact same signed application APKs built via Nix across multiple devices.

I've also recently pushed some major revisions to the documentation https://github.com/danielfullmer/robotnix/pull/88 that will hopefully be a bit more helpful. If you find more places where we could improve the new-user experience please feel free to contribute :)

codethief commented 3 years ago

Hi Daniel, thank you so much for this detailed response! It helps a lot! :)

You set signing.enable = true; if you want to have signed builds, no matter whether you use -A img or -A releaseScript.

Looks like I'll have to rebuild everything again then haha. :( I had thought that -A releaseScript with signing.enable = false would still result in a properly signed build once I do ./release.sh ./keys. (Interestingly enough, I was able to flash it, lock the bootloader etc. and everything seemed to work.)

danielfullmer commented 3 years ago

@codethief Running ./release.sh ./keys should correctly sign your build with your own keys. However, if you are using any robotnix modules which need information about the keys/certificate fingerprints (IIRC just F-Droid and Auditor), those modules will be using incorrect information if you had not set signing.enable = true; / keyStorePath = "..." or manually set things like signing.avb.fingerprint. I don't think that should be a security problem, but those individual apps (F-Droid and Auditor) might not work correctly.

Added warning in 28c8826cfb401b0fb3bfe092e71900b6c4956592

danielfullmer commented 3 years ago

The recent updates to https://docs.robotnix.org/ should address the questions asked so far, so I'll close this for now. If you have additional questions feel free to continue to ask here or on #robotnix on Freenode.

codethief commented 3 years ago

@danielfullmer The new docs look really good, thank you so much! :)