[RFC] Use Nix to build initrd

YanWQ-monad commented 5 months ago

Summary

Use Nix build system to build initrd image (and it can also be used to manage the development environment).

Motivation

Currently, Asterinas uses Makefile to build initrd, but with the following drawbacks.

Hard to support cross-compilation. As we are introducing RISC-V support, so our initrd builder should be able to build initrd of different architectures. In current Makefile system, we need to add cross compiler in asterinas Docker image, and add corresponding config in Makefile, which may be increasingly hard to maintain if we introduce ARM and other architectures.
Hard to build complex software. For example, to build Python, we need to install the dependencies like libgdbm-dev, libgdbm-compat-dev, and etc, and copy necessary shared objects to initrd. The shared objects are manually chosen and are hard to maintain. What's worse, in cross compilation, we need to build all the dependencies by ourselves, because it can't simply be installed by apt-get, and this could be a recursion hell.
Imperative. Currently, The recipe to build initrd is imperative. This could cause some problems. For example, for the recipe
```
$(INITRAMFS)/etc:
   @mkdir -p $@
   @cp $(CUR_DIR)/etc/passwd $@
   @cp $(CUR_DIR)/etc/group $@
```
if interrupt it when cping passwd, and run make again, Makefile will skip this recipe since /etc exists, despite its content is not properly created.

Besides, current Makefile contains lots of hardcoded paths, like VDSO_DIR := /root/dependency and
```
$(INITRAMFS)/lib/x86_64-linux-gnu: | $(VDSO_LIB)
   @mkdir -p $@
   @cp -L /lib/x86_64-linux-gnu/libc.so.6 $@
   @cp -L /lib/x86_64-linux-gnu/libstdc++.so.6 $@
```
which could make it hard to build initrd in a more general Linux environment.

So we need a another build system to build initrd that has better cross compilation support, and can manage initrd's content in a more clear way.

Proposal

Nix is a package manager, and the introduction is on https://nix.dev/manual/nix/2.18/.

Some background or brief introduction about Nix

Nix has a special methodology to manage packages, all packages' content will be put in the corresponding directory in /nix/store, for example, /nix/store/fif0jnmkic89znf57sng621zmg4w71c2-busybox-1.36.1 for a busybox, where fif0j...71c2 is the hash of the build configuration (if we change the build configuration, the hash will change so a new package will be created), busybox is the name of the package, and 1.36.1 is the version. And inside /nix/store/fif0jnmkic89znf57sng621zmg4w71c2-busybox-1.36.1, all files are read-only. Its dependent shared objects are explicitly set to another package, for example,
$ ldd /nix/store/fif0jnmkic89znf57sng621zmg4w71c2-busybox-1.36.1/bin/busybox
      linux-vdso.so.1 (0x00007ffc42bc0000)
      libm.so.6 => /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libm.so.6 (0x00007f3e68ba8000)
      libresolv.so.2 => /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libresolv.so.2 (0x00007f3e68b97000)
      libc.so.6 => /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libc.so.6 (0x00007f3e689aa000)
      /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/ld-linux-x86-64.so.2 => /nix/store/dbwp0scbb0rk78m636sb7cvycz8xzgyh-glibc-2.39-52/lib64/ld-linux-x86-64.so.2 (0x00007f3e68c8d000)
So a package is immutable once it's built, and it won't be broken by packages or dependencies upgrade, because when upgrading, a new package will be built instead of replacing the existing one.

And to use a package, instead of installing it to the system's /usr/local/bin, we just add the package's path to PATH. To use multiple packages, we usually create a "environment" which consists of symbolic links to the package we want to use, as described in https://nix.dev/manual/nix/2.18/package-management/profiles.

And I am currently using Nix to manage user packages in my machine, and using system manager to manage system packages in my machine. It can reduce the risk of breaking system when installing/upgrading packages, and breaking user packages when upgrading the system.

With Nix, a basic initrd with only busybox can be build with the following Nix script:

{
  description = "Asterinas Initrd";
  inputs.nixpkgs.url = "nixpkgs/nixos-unstable";

  outputs = { self, nixpkgs }: {
    packages."x86_64-linux" = let
      pkgs = import nixpkgs { system = "x86_64-linux"; };
    in rec {
      initrd = pkgs.makeInitrdNG {
        name = "aster-initrd";
        compressor = "gzip";

        contents = let
          initrdBinEnv = pkgs.buildEnv {
            name = "initrd-bin-env";
            paths = [
              # The packages we want to use in initrd
              pkgs.busybox
            ];
            pathsToLink = ["/bin"];
          };

          contents = {
            # Create directories for mountpoints
            "/dev/.keep".text = "/dev mount point";
            "/proc/.keep".text = "/proc mount point";
            "/ext2/.keep".text = "/ext2 mount point";
            "/exfat/.keep".text = "/exfat mount point";

            # Link `/bin` to the environment's `/bin`
            "/bin".source = "${initrdBinEnv}/bin";

            # ./vdso64.so without quotes is to use a file in the current directory
            "/lib/x86_64-linux-gnu/vdso64.so".source = ./vdso64.so;
          };

          # The packages needed to add to `/nix/store` but not needed to be symbolic linked
          storePaths = [
          ];

          # A little stuff for adapter
          canonicalizeContent = symlink: v: if builtins.hasAttr "text" v
            then builtins.toFile (builtins.baseNameOf symlink) v.text
            else v.source;
        in
          map (path: { object = path; symlink = null; }) storePaths
          ++ pkgs.lib.mapAttrsToList (k: v: { object = canonicalizeContent k v; symlink = k; }) contents;
      };

      default = initrd;
    };
  };
}

And then Nix will build all the dependencies and then copy binaries and necessary shared objects to initrd. This dependencies building and resolving are done by Nix, without human intervention.

To build RISC-V initrd, we can replace

pkgs = import nixpkgs { system = "x86_64-linux"; };

with

pkgs = (import nixpkgs { system = "x86_64-linux"; }).pkgsCross.riscv64;

then it will build RISC-V's packages.

What's more, using Nix can make it easy to add packages to initrd. For example, adding Python only needs to add pkgs.python3 in pkgs.buildEnv and storePaths, and it works. We don't need to care about how to recursively build its dependencies.

Side Note: Asterinas can run basic Python scripts (complex ones are not tested yet), but Python REPL is unusable because pselect6 syscall is not supported yet.

Besides initrd, with nix-direnv, Nix can also be used to setup a local development environment for Asterinas. This could be better than Docker because Docker is too isolated, it's suitable for deployment but not development, while using Nix can reuse our friendly shell.

Known Issues

Currently syscall tests are built with Bazel, but both Nix and Bazel are "complete" systems, they don't cooperate well. The issue itself can be solved by patching gVisor, but it's not elegant. What's more, gVisor's syscall tests seem don't support RISC-V architecture (though there some "RISC-V" pieces in its build scripts, the README says "_gVisor builds on x8664 and ARM64"). But how to deal with syscall tests with RISC-V may be beyond the scope of this RFC.
The build unit in Nix is package, so it doesn't support increment build very well currently. That is, if one modifies a file of a package, Nix will build the whole package again instead of the changed ones. This issue may be mitigated by building the package manually and copying the binaries in Nix script.

kylerky commented 2 months ago

Nix has a package for gVisor. It is very likely that you just need to change the package definition to use the forked repository. For RISC-V compatibility, it is true that you will need to apply some patches. I think Nix should have the required mechanisms in place. Is that the case?

YanWQ-monad commented 2 months ago

Nix has a package for gVisor.

This package is based on the go branch of gVisor, which only contains the necessary files to build gVisor binary, and other files such as bazel configurations and syscall tests code are missing, thus the branch can be used to build gVisor without bazel.
This approach is OK if one only wants to build the gVisor binary. But we want its syscall tests, which are built by bazel, so this package seems cannot be reused.

Reference: https://github.com/NixOS/nixpkgs/blob/d31b1025834fea78b31ff7b2a57cc2060d93a5bd/pkgs/by-name/gv/gvisor/package.nix#L16-L18

asterinas / asterinas