search

DeterminateSystems / nix-installer

Install Nix and flakes with the fast and reliable Determinate Nix Installer, with over 2 million installs.
GNU Lesser General Public License v2.1
2.04k stars 52 forks source link

permission errors on `/nix` when running from initrd (busybox) #446

Open tlaurion opened 1 year ago

tlaurion commented 1 year ago

Error

Error: 
   0: Install failure
   1: Error executing action
   2: Action `configure_nix` errored
   3: Action `setup_default_profile` errored
   4: Failed to execute command with status 100 `HOME="/" NIX_SSL_CERT_FILE="/nix/store/ba4f8msp39cfvfpw3m7fsalb4psw347z-nss-cacert-3.83/etc/ssl/certs/ca-bundle.crt" "/nix/store/mc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3/bin/nix-env" "-i" "/nix/store/mc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3"`, stdout: 
      stderr: installing 'nix-2.13.3'
      building '/nix/store/3yc97yq64yrs0mjxy8hsdrzn9fk2np3r-user-environment.drv'...
      error: creating directory '/nix': Permission denied
      error: builder for '/nix/store/3yc97yq64yrs0mjxy8hsdrzn9fk2np3r-user-environment.drv' failed with exit code 1

Metadata

key value
version 0.8.0
os linux
arch x86_64

Backtrace

``` Backtrace: 0: 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: ```

Not sure what else needs to be in under busybox at this point and think this is a bug? My understanding of launching the nix-installer with RUST_BACKTRACE=full /tmp/nix-installer install linux --extra-conf "sandbox = false" --init none was that neither cgroups nor addgroup/adduser/deluser/delgroup were needed, nut I added them in my PoC nevertheless (change of kernel config as per #438 and documented on screen per installer) but now I'm a bit in the dark.

I am not sure of what is happening here:

Nix install plan (v0.8.0)
Planner: linux

Configured settings:
* extra_conf: ["sandbox = false"]
* init: "None"
* start_daemon: true

Planned actions:
* Create directory `/nix`
* Fetch `https://releases.nixos.org/nix/nix-2.13.3/nix-2.13.3-x86_64-linux.tar.xz` to `/nix/temp-install-dir`
* Create build users (UID 30000-30032) and group (GID 30000)
* Create a directory tree in `/nix`
* Move the downloaded Nix into `/nix`
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
* Configure the shell profiles
* Remove directory `/nix/temp-install-dir`

Proceed? ([Y]es/[n]o/[e]xplain): y
 INFO Step: Create directory `/nix`
 INFO Step: Provision Nix
 INFO Step: Configure Nix

This is the permissions of current state of /nix

~ # ls /nix/* -al
-rwxr-xr-x    1 root     root       8318008 May  2 14:23 /nix/nix-installer
-rw-r--r--    1 root     root         29085 May  2 14:23 /nix/receipt.json

/nix/store:
drwxrwxr-t   47 root     nixbld        1000 May  2 14:23 .
drwxr-xr-x    5 root     root           140 May  2 14:23 ..
drwxr-xr-x    2 root     root            80 May  2 14:23 .links
drwxr-xr-x    4 root     root            80 May  2 14:23 026hln0aq1hyshaxsdvhg0kmcm6yf45r-zlib-1.2.13
drwxr-xr-x    4 root     root            80 May  2 14:23 1f3pdcihv4mmgfsyw4jxqjvn7j5sc2xy-aws-c-mqtt-0.7.13
drwxr-xr-x    5 root     root           100 May  2 14:23 20d2d0jap81kxp5ng1igl7kkgnd077f3-libcpuid-0.6.2
drwxr-xr-x    3 root     root            60 May  2 14:23 34xlpp3j3vy7ksn09zh44f1c04w77khf-libunistring-1.0
drwxr-xr-x    5 root     root           100 May  2 14:23 3j1h6psl4pzn6b3yck6rk33bpwrmihb1-aws-c-common-0.8.5
drwxr-xr-x    3 root     root            60 May  2 14:23 3n9vwzn9wkq7cj93jp0s6gqpx1zbhvlj-libarchive-3.6.1-lib
drwxr-xr-x    4 root     root            80 May  2 14:23 3rj4dwafjii1wi2f5va26hindims9zyz-boehm-gc-8.2.2
-r--r--r--    2 root     root           546 Jan  1  1970 3yc97yq64yrs0mjxy8hsdrzn9fk2np3r-user-environment.drv
drwxr-xr-x    4 root     root            80 May  2 14:23 3z091ijyij0mmgi3iv0mp945lm2bx4wy-aws-checksums-0.1.13
drwxr-xr-x    4 root     root            80 May  2 14:23 4mxnw95jcm5a27qk60z7yc0gvxp42b9a-openssl-3.0.7
drwxr-xr-x    6 root     root           140 May  2 14:23 4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163
drwxr-xr-x    3 root     root            60 May  2 14:23 4rkhsf7sig2lh303bygqr3ph5mfwz0ah-s2n-tls-1.3.28
drwxr-xr-x    4 root     root            80 May  2 14:23 50si0kkawnfkgs8m7d8iv2zmkq0fdbm7-aws-c-event-stream-0.2.15
drwxr-xr-x    4 root     root            80 May  2 14:23 5glq0q102jva5p0h9ap0f1j9ps24gcc0-aws-sdk-cpp-1.9.294
drwxr-xr-x    4 root     root            80 May  2 14:23 5mh5019jigj0k14rdnjam1xwk5avn1id-libidn2-2.3.2
drwxr-xr-x    3 root     root            60 May  2 14:23 5q73izqsdasz81gma1nz6870b6220vkh-aws-crt-cpp-0.18.9
drwxr-xr-x    4 root     root            80 May  2 14:23 6qqybxxz6636jymz8x7l2sxj4np9yzsy-aws-c-cal-0.5.20
drwxr-xr-x    3 root     root            80 May  2 14:23 7b943a2k4amjmam6dnwnxnj8qbba9lbq-busybox-static-x86_64-unknown-linux-musl-1.35.0
drwxr-xr-x    4 root     root            80 May  2 14:23 7q41sbf04qcwv75j5bxis6pfjnmshy44-acl-2.3.1
drwxr-xr-x    3 root     root            60 May  2 14:23 816qwr4xy058451rbxr0ccyh1v1akhb6-keyutils-1.6.3-lib
drwxr-xr-x    4 root     root            80 May  2 14:23 84px25a6dsmdg7ni7186rmkybq1k2vrf-aws-c-s3-0.1.51
drwxr-xr-x    3 root     root            60 May  2 14:23 9iy1ng7h1l6jdmjk157jra8n4hkrfdj1-brotli-1.0.9-lib
drwxr-xr-x    4 root     root            80 May  2 14:23 9x6jfl84234im3f4gsrl1js3zyqz76jd-nlohmann_json-3.11.2
drwxr-xr-x    4 root     root            80 May  2 14:23 a6q74vrxbsrmmw1z0fdzd0qrcm6dc0ll-aws-c-http-0.6.27
drwxr-xr-x    3 root     root            60 May  2 14:23 b7ac47isdc4z1ajln6adz3ppcjs1708b-libsodium-1.0.18
drwxr-xr-x    4 root     root            80 May  2 14:23 ba4f8msp39cfvfpw3m7fsalb4psw347z-nss-cacert-3.83
drwxr-xr-x    5 root     root           100 May  2 14:23 bwsyv47ri47ppr4ga34wd2khk89ch4n4-aws-c-auth-0.6.21
drwxr-xr-x    3 root     root            60 May  2 14:23 c8byvs0rj8vg5cpm5mswcg5dvp7d5ir7-libseccomp-2.5.4-lib
drwxr-xr-x    5 root     root           100 May  2 14:23 cr5fmwri3601s7724ayjvckhsg6cz4rv-attr-2.5.1
drwxr-xr-x    4 root     root            80 May  2 14:23 dsd5gz46hdbdk2rfdimqddhq6m8m8fqs-bash-5.1-p16
drwxr-xr-x    4 root     root            80 May  2 14:23 f116ly4fl72zc5ynb03qrwdk2bp8y61p-aws-c-io-0.13.11
drwxr-xr-x    4 root     root            80 May  2 14:23 fph0r1qjhsygrb420y2zsfhjh3rssq0z-aws-c-sdkutils-0.1.7
drwxr-xr-x    3 root     root            60 May  2 14:23 h5slhj7gqpqh5q4jb00xdbcwbl8vqqa7-libxml2-2.10.3
drwxr-xr-x    3 root     root            60 May  2 14:23 il7dydwf1wfn9b4p4cii8q99b93i106n-editline-1.17.1
drwxr-xr-x    3 root     root            60 May  2 14:23 jpj9lx0p2h1vs3gkzj8jh350113bsm84-sqlite-3.39.4
drwxr-xr-x    7 root     root           140 May  2 14:23 mc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3
drwxr-xr-x    4 root     root           100 May  2 14:23 mdck89nsfisflwjv6xv8ydj7dj0sj2pn-gcc-11.3.0-lib
-r--r--r--    2 root     root           229 Jan  1  1970 p3hdf6aya5rj8z5g4m0m6ll2zbj2g7jw-env-manifest.nix
-rw-------    1 root     root             0 May  2 14:23 pzvif1ab1n641vg55zhnha2ipsb6qpcc-user-environment.lock
drwxr-xr-x    3 root     root            60 May  2 14:23 qz400bwshaqikj5s2qyvh0c9qffgmqik-nghttp2-1.49.0-lib
drwxr-xr-x    5 root     root           120 May  2 14:23 r7gl900my2fw6k33nxh2r7rzv8nv0s25-libkrb5-1.20
drwxr-xr-x    4 root     root            80 May  2 14:23 rirzp6ijbcwnxlf0b2n286n587r3z9jw-curl-7.86.0
drwxr-xr-x    3 root     root            60 May  2 14:23 vqq9s0d6fw6kqf3sr5nrzqbys9rhygqd-libssh2-1.10.0
drwxr-xr-x    5 root     root           100 May  2 14:23 w10in9diaqrcqqxi5lg20n3q2jfpk6pq-zstd-1.5.2
drwxr-xr-x    4 root     root            80 May  2 14:23 w3sdhqiazzp4iy40wc2g85mv0grg1cx0-xz-5.2.7
drwxr-xr-x    4 root     root            80 May  2 14:23 wnxn8173p8gj888wwgv2l9czp7zf4jl3-aws-c-compression-0.2.16
drwxr-xr-x    3 root     root            60 May  2 14:23 ysl6qj5r7nn63b16954dhk7x47r5yq7i-bzip2-1.0.8

/nix/temp-install-dir:
drwxr-xr-x    3 root     root            60 May  2 14:23 .
drwxr-xr-x    5 root     root           140 May  2 14:23 ..
drwxr-xr-x    3 root     root           180 May  2 14:23 nix-2.13.3-x86_64-linux

/nix/var:
drwxr-xr-x    4 root     root            80 May  2 14:23 .
drwxr-xr-x    5 root     root           140 May  2 14:23 ..
drwxr-xr-x    3 root     root            60 May  2 14:23 log
drwxr-xr-x    8 root     root           180 May  2 14:23 nix
tlaurion commented 1 year ago

Ideally, no new groups nor new users would be added and root should be usable to deal with the store on demand in my use case (we are under initramfs attempting to have a clean downloaded and up to date Nix installation in ram, and maybe later on deployed on disk if user decides to keep state persistent.

This is why I was referred here afterall: --extra-conf "sandbox = false" --init none was offering the promise of being stateless and permitting to install chached, trustable binaries.

Am I missing something?

tlaurion commented 1 year ago

@Hoverbear any insights?

Hoverbear commented 1 year ago

Based on https://github.com/DeterminateSystems/nix-installer/issues/446#issuecomment-1531648944 it seems like you are running in a fairly exotic situation.

It's rather curious because at the point you experienced a failure we've already created /nix, indeed you can even see it is created in https://github.com/DeterminateSystems/nix-installer/issues/446#issue-1692533752, so I don't understand why we'd see that error.

Is it possible for me to easily reproduce this?

Ideally, no new groups nor new users would be added

At this time we don't support a single user install like this. I suggest looking at the single user option of the install scripts: https://nixos.org/manual/nix/stable/installation/installing-binary.html#single-user-installation

tlaurion commented 1 year ago

At this time we don't support a single user install like this. I suggest looking at the single user option of the install scripts: https://nixos.org/manual/nix/stable/installation/installing-binary.html#single-user-installation

As of now, my really basic PoC is at https://github.com/tlaurion/heads/tree/staging_all Heads is single user system, based on coreboot+linux+busybox. Now present testing config includes busybox support for adduser/deluser/groupadd/groupdel which should not be used on upstream --no-deamon invocation (if I understand well).

Following your comment, I tested upstream instructions to not launch daemon and expect single user system, even if root is not supposed to be supported, meaning that separation of duties would not be supported:

~ # wget https://nixos.org/nix/install -O /tmp/nix-installer
[  131.796744] random: crng init done
[  131.803283] random: 1 urandom warning(s) missed due to ratelimiting
~ # chmod u+x /tmp/nix-installer 
~ # mkdir /nix
~ # /tmp/nix-installer --no-daemon
downloading Nix 2.15.0 binary tarball for x86_64-linux from 'https://releases.nixos.org/nix/nix-2.15.0/nix-2.15.0-x86_64-linux.tar.xz' to '/tmp/nix-binary-tarball-unpack.XXXXfGaagf'...
Note: a multi-user installation is possible. See https://nixos.org/manual/nix/stable/installation/installing-binary.html#multi-user-installation
warning: installing Nix as root is not supported by this script!
performing a single-user installation of Nix...
copying Nix to /nix/store.................................................
warning: the group 'nixbld' specified in 'build-users-group' does not exist
warning: the group 'nixbld' specified in 'build-users-group' does not exist
installing 'nix-2.15.0'
error: the group 'nixbld' specified in 'build-users-group' does not exist
/tmp/nix-binary-tarball-unpack.XXXXfGaagf/unpack/nix-2.15.0-x86_64-linux/install: unable to install Nix into your default profile

I'm not sure what I do not understand here, but single user installation should not involve groups....

tlaurion commented 1 year ago

Getting back to this isssue related to this project, I can probably give more information on environment.

~ # busybox --help
BusyBox v1.36.0 (heads) multi-call binary.
BusyBox is copyrighted by many authors between 1998-2015.
Licensed under GPLv2. See source distribution for detailed
copyright notices.

Usage: busybox [function [arguments]...]
   or: busybox --list
   or: function [arguments]...

    BusyBox is a multi-call binary that combines many common Unix
    utilities into a single executable.  Most people will create a
    link to busybox for each function they wish to use and BusyBox
    will act like whatever it was invoked as.

Currently defined functions:
    [, [[, addgroup, adduser, arch, arp, ascii, ash, awk, base32, basename,
    blkid, blockdev, bunzip2, bzcat, bzip2, cat, chattr, chmod, chroot,
    cmp, cp, cpio, crc32, cut, date, dc, dd, delgroup, deluser, devmem, df,
    diff, dirname, dmesg, du, echo, env, expr, factor, fallocate, false,
    fdisk, find, fold, fsck, fsfreeze, getopt, grep, groups, gunzip, gzip,
    hd, head, hexdump, hexedit, hostid, hwclock, i2cdetect, i2cdump,
    i2cget, i2cset, id, ifconfig, insmod, install, ip, kill, killall,
    killall5, less, link, ln, loadkmap, losetup, ls, lsattr, lsmod, lsof,
    lsscsi, lsusb, md5sum, mkdir, mkdosfs, mke2fs, mkfifo, mkfs.vfat,
    mknod, mktemp, modinfo, more, mount, mv, nc, nl, nproc, nslookup, ntpd,
    partprobe, paste, patch, pgrep, pidof, ping, pkill, printf, ps, pwd,
    readlink, realpath, resume, rm, rmdir, route, sed, seedrng, seq,
    setfattr, setpriv, setserial, setsid, sh, sha1sum, sha256sum, sha3sum,
    sha512sum, shred, sleep, sort, ssl_client, stat, strings, stty, sync,
    sysctl, tail, tar, tee, test, tftp, time, top, touch, tr, tree, true,
    tsort, tty, udhcpc, umount, uname, uniq, unxz, unzip, usleep, vconfig,
    vi, wc, wget, which, xargs, xxd, xz, xzcat, zcat

Everything runs in root unless modifications applied on top of running system. This is where I would love to know more on what happens between the two steps that fails above.

Agreeed, this is confusing but I think we might miss a more verbose output in master to see what is going wrong between those two steps.

I want to bring your attention on above output:

/nix/store: drwxrwxr-t 47 root nixbld 1000 May 2 14:23 .

All other permissions are setuped to root.

tlaurion commented 1 year ago

Redoing with fullest output available.

#removing busybox wget to use u-root with ca under /etc/ssl/certs:
~ # rm /bin/wget
~ # /bbin/wget 
2023/05/03 20:52:33 Usage: /bbin/wget [ARGS] URL
~ # wget https://install.determinate.systems/nix -O /tmp/nix-installer
~ # chmod +x /tmp/nix-installer
~ # /tmp/nix-installer install linux --extra-conf "sandbox = false" --init none
info: downloading installer https://install.determinate.systems/nix/tag/v0.8.0/nix-installer-x86_64-linux
flag provided but not defined: -V
Usage of wget:
  -O string
        output file
2023/05/03 20:54:26 flag provided but not defined: -V
flag provided but not defined: -V
Usage of wget:
  -O string
        output file
2023/05/03 20:54:26 flag provided but not defined: -V
Warning: Not enforcing strong cipher suites for TLS, this is potentially less secure
Usage of wget:
  -O string
        output file
2023/05/03 20:54:27 flag: help requested
Usage of wget:
  -O string
        output file
2023/05/03 20:54:27 flag: help requested
Warning: Not enforcing TLS v1.2, this is potentially less secure
Nix install plan (v0.8.0)
Planner: linux

Configured settings:
* extra_conf: ["sandbox = false"]
* init: "None"
* start_daemon: true

Planned actions:
* Create directory `/nix`
* Fetch `https://releases.nixos.org/nix/nix-2.13.3/nix-2.13.3-x86_64-linux.tar.xz` to `/nix/temp-install-dir`
* Create build users (UID 30000-30032) and group (GID 30000)
* Create a directory tree in `/nix`
* Move the downloaded Nix into `/nix`
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
* Configure the shell profiles
* Remove directory `/nix/temp-install-dir`

Proceed? ([Y]es/[n]o/[e]xplain): e
Nix install plan (v0.8.0)
Planner: linux

Configured settings:
* diagnostic_endpoint: "https://install.determinate.systems/nix/diagnostic"
* extra_conf: ["sandbox = false"]
* force: false
* init: "None"
* modify_profile: true
* nix_build_group_id: 30000
* nix_build_group_name: "nixbld"
* nix_build_user_count: 32
* nix_build_user_id_base: 30000
* nix_build_user_prefix: "nixbld"
* nix_package_url: "https://releases.nixos.org/nix/nix-2.13.3/nix-2.13.3-x86_64-linux.tar.xz"
* proxy: null
* ssl_cert_file: null
* start_daemon: true

Planned actions:
* Create directory `/nix`
* Fetch `https://releases.nixos.org/nix/nix-2.13.3/nix-2.13.3-x86_64-linux.tar.xz` to `/nix/temp-install-dir`
* Create build users (UID 30000-30032) and group (GID 30000)
  The Nix daemon requires system users (and a group they share) which it can act as in order to build
  Create group `nixbld` (GID 30000)
  Create user `nixbld1` (UID 30001) in group `nixbld` (GID 30000)
  Create user `nixbld2` (UID 30002) in group `nixbld` (GID 30000)
  Create user `nixbld3` (UID 30003) in group `nixbld` (GID 30000)
  Create user `nixbld4` (UID 30004) in group `nixbld` (GID 30000)
  Create user `nixbld5` (UID 30005) in group `nixbld` (GID 30000)
  Create user `nixbld6` (UID 30006) in group `nixbld` (GID 30000)
  Create user `nixbld7` (UID 30007) in group `nixbld` (GID 30000)
  Create user `nixbld8` (UID 30008) in group `nixbld` (GID 30000)
  Create user `nixbld9` (UID 30009) in group `nixbld` (GID 30000)
  Create user `nixbld10` (UID 30010) in group `nixbld` (GID 30000)
  Create user `nixbld11` (UID 30011) in group `nixbld` (GID 30000)
  Create user `nixbld12` (UID 30012) in group `nixbld` (GID 30000)
  Create user `nixbld13` (UID 30013) in group `nixbld` (GID 30000)
  Create user `nixbld14` (UID 30014) in group `nixbld` (GID 30000)
  Create user `nixbld15` (UID 30015) in group `nixbld` (GID 30000)
  Create user `nixbld16` (UID 30016) in group `nixbld` (GID 30000)
  Create user `nixbld17` (UID 30017) in group `nixbld` (GID 30000)
  Create user `nixbld18` (UID 30018) in group `nixbld` (GID 30000)
  Create user `nixbld19` (UID 30019) in group `nixbld` (GID 30000)
  Create user `nixbld20` (UID 30020) in group `nixbld` (GID 30000)
  Create user `nixbld21` (UID 30021) in group `nixbld` (GID 30000)
  Create user `nixbld22` (UID 30022) in group `nixbld` (GID 30000)
  Create user `nixbld23` (UID 30023) in group `nixbld` (GID 30000)
  Create user `nixbld24` (UID 30024) in group `nixbld` (GID 30000)
  Create user `nixbld25` (UID 30025) in group `nixbld` (GID 30000)
  Create user `nixbld26` (UID 30026) in group `nixbld` (GID 30000)
  Create user `nixbld27` (UID 30027) in group `nixbld` (GID 30000)
  Create user `nixbld28` (UID 30028) in group `nixbld` (GID 30000)
  Create user `nixbld29` (UID 30029) in group `nixbld` (GID 30000)
  Create user `nixbld30` (UID 30030) in group `nixbld` (GID 30000)
  Create user `nixbld31` (UID 30031) in group `nixbld` (GID 30000)
  Create user `nixbld32` (UID 30032) in group `nixbld` (GID 30000)
  Add user `nixbld1` (UID 30001) to group `nixbld` (GID 30000)
  Add user `nixbld2` (UID 30002) to group `nixbld` (GID 30000)
  Add user `nixbld3` (UID 30003) to group `nixbld` (GID 30000)
  Add user `nixbld4` (UID 30004) to group `nixbld` (GID 30000)
  Add user `nixbld5` (UID 30005) to group `nixbld` (GID 30000)
  Add user `nixbld6` (UID 30006) to group `nixbld` (GID 30000)
  Add user `nixbld7` (UID 30007) to group `nixbld` (GID 30000)
  Add user `nixbld8` (UID 30008) to group `nixbld` (GID 30000)
  Add user `nixbld9` (UID 30009) to group `nixbld` (GID 30000)
  Add user `nixbld10` (UID 30010) to group `nixbld` (GID 30000)
  Add user `nixbld11` (UID 30011) to group `nixbld` (GID 30000)
  Add user `nixbld12` (UID 30012) to group `nixbld` (GID 30000)
  Add user `nixbld13` (UID 30013) to group `nixbld` (GID 30000)
  Add user `nixbld14` (UID 30014) to group `nixbld` (GID 30000)
  Add user `nixbld15` (UID 30015) to group `nixbld` (GID 30000)
  Add user `nixbld16` (UID 30016) to group `nixbld` (GID 30000)
  Add user `nixbld17` (UID 30017) to group `nixbld` (GID 30000)
  Add user `nixbld18` (UID 30018) to group `nixbld` (GID 30000)
  Add user `nixbld19` (UID 30019) to group `nixbld` (GID 30000)
  Add user `nixbld20` (UID 30020) to group `nixbld` (GID 30000)
  Add user `nixbld21` (UID 30021) to group `nixbld` (GID 30000)
  Add user `nixbld22` (UID 30022) to group `nixbld` (GID 30000)
  Add user `nixbld23` (UID 30023) to group `nixbld` (GID 30000)
  Add user `nixbld24` (UID 30024) to group `nixbld` (GID 30000)
  Add user `nixbld25` (UID 30025) to group `nixbld` (GID 30000)
  Add user `nixbld26` (UID 30026) to group `nixbld` (GID 30000)
  Add user `nixbld27` (UID 30027) to group `nixbld` (GID 30000)
  Add user `nixbld28` (UID 30028) to group `nixbld` (GID 30000)
  Add user `nixbld29` (UID 30029) to group `nixbld` (GID 30000)
  Add user `nixbld30` (UID 30030) to group `nixbld` (GID 30000)
  Add user `nixbld31` (UID 30031) to group `nixbld` (GID 30000)
  Add user `nixbld32` (UID 30032) to group `nixbld` (GID 30000)
* Create a directory tree in `/nix`
  Create directory `/nix/var`
  Create directory `/nix/var/log`
  Create directory `/nix/var/log/nix`
  Create directory `/nix/var/log/nix/drvs`
  Create directory `/nix/var/nix`
  Create directory `/nix/var/nix/db`
  Create directory `/nix/var/nix/gcroots`
  Create directory `/nix/var/nix/gcroots/per-user`
  Create directory `/nix/var/nix/profiles`
  Create directory `/nix/var/nix/profiles/per-user`
  Create directory `/nix/var/nix/temproots`
  Create directory `/nix/var/nix/userpool`
  Create directory `/nix/var/nix/daemon-socket`
* Move the downloaded Nix into `/nix`
  Nix is being downloaded to `/nix/temp-install-dir` and should be in `/nix`
* Setup the default Nix profile
* Place the Nix configuration in `/etc/nix/nix.conf`
  This file is read by the Nix daemon to set its configuration options at runtime.
  Create directory `/etc/nix`
  Merge or create nix.conf file `/etc/nix/nix.conf`
* Configure the shell profiles
  Update shell profiles to import Nix
* Remove directory `/nix/temp-install-dir`

Proceed? ([Y]es/[n]o): y
 INFO Step: Create directory `/nix`
 INFO Step: Provision Nix
 INFO Step: Configure Nix
ERROR 
   0: Install failure
   1: Error executing action
   2: Action `configure_nix` errored
   3: Action `setup_default_profile` errored
   4: Failed to execute command with status 100 `HOME="/" NIX_SSL_CERT_FILE="/nix/store/ba4f8msp39cfvfpw3m7fsalb4psw347z-nss-cacert-3.83/etc/ssl/certs/ca-bundle.crt" "/nix/store/mc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3/bin/nix-env" "-i" "/nix/store/mc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3"`, stdout: 
      stderr: installing 'nix-2.13.3'
      building '/nix/store/3yc97yq64yrs0mjxy8hsdrzn9fk2np3r-user-environment.drv'...
      error: creating directory '/nix': Permission denied
      error: builder for '/nix/store/3yc97yq64yrs0mjxy8hsdrzn9fk2np3r-user-environment.drv' failed with exit code 1

   4: 

Location:
   /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/convert/mod.rs:726

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Consider reporting this error using this URL: https://github.com/DeterminateSystems/nix-installer/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0AError%3A+%0A+++0%3A+Install+failure%0A+++1%3A+Error+executing+action%0A+++2%3A+Action+%60configure_nix%60+errored%0A+++3%3A+Action+%60setup_default_profile%60+errored%0A+++4%3A+Failed+to+execute+command+with+status+100+%60HOME%3D%22%2F%22+NIX_SSL_CERT_FILE%3D%22%2Fnix%2Fstore%2Fba4f8msp39cfvfpw3m7fsalb4psw347z-nss-cacert-3.83%2Fetc%2Fssl%2Fcerts%2Fca-bundle.crt%22+%22%2Fnix%2Fstore%2Fmc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3%2Fbin%2Fnix-env%22+%22-i%22+%22%2Fnix%2Fstore%2Fmc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3%22%60%2C+stdout%3A+%0A++++++stderr%3A+installing+%27nix-2.13.3%27%0A++++++building+%27%2Fnix%2Fstore%2F3yc97yq64yrs0mjxy8hsdrzn9fk2np3r-user-environment.drv%27...%0A++++++error%3A+creating+directory+%27%2Fnix%27%3A+Permission+denied%0A++++++error%3A+builder+for+%27%2Fnix%2Fstore%2F3yc97yq64yrs0mjxy8hsdrzn9fk2np3r-user-environment.drv%27+failed+with+exit+code+1%0A%0A%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7C**version**%7C0.8.0%7C%0A%7C**os**%7Clinux%7C%0A%7C**arch**%7Cx86_64%7C%0A
Installation failure, offering to revert...
Nix uninstall plan (v0.8.0)

Planner: linux

Configured settings:
* extra_conf: ["sandbox = false"]
* init: "None"
* start_daemon: true

Planned actions:
* Unconfigure the shell profiles
* Remove the Nix configuration in `/etc/nix/nix.conf`
* Unset the default Nix profile
* Remove the directory tree in `/nix`
* Remove Nix users and group
* Remove the directory `/nix`

Proceed? ([Y]es/[n]o/[e]xplain): 
Proceed? ([Y]es/[n]o/[e]xplain): n
Okay, didn't do anything! Bye!
~ # cat /etc/nix/nix.conf 
# Generated by https://github.com/DeterminateSystems/nix-installer, version 0.8.0.
sandbox = false
build-users-group = nixbld
experimental-features = nix-command flakes
bash-prompt-prefix = (nix:$name)\040
auto-optimise-store = true
extra-nix-path = nixpkgs=flake:nixpkgs
~ # cat /etc/passwd
root:x:0:0:root:/:/bin/sh
nixbld1:x:30001:30000:Nix build user 1:/var/empty:/sbin/nologin
nixbld2:x:30002:30000:Nix build user 2:/var/empty:/sbin/nologin
nixbld3:x:30003:30000:Nix build user 3:/var/empty:/sbin/nologin
nixbld4:x:30004:30000:Nix build user 4:/var/empty:/sbin/nologin
nixbld5:x:30005:30000:Nix build user 5:/var/empty:/sbin/nologin
nixbld6:x:30006:30000:Nix build user 6:/var/empty:/sbin/nologin
nixbld7:x:30007:30000:Nix build user 7:/var/empty:/sbin/nologin
nixbld8:x:30008:30000:Nix build user 8:/var/empty:/sbin/nologin
nixbld9:x:30009:30000:Nix build user 9:/var/empty:/sbin/nologin
nixbld10:x:30010:30000:Nix build user 10:/var/empty:/sbin/nologin
nixbld11:x:30011:30000:Nix build user 11:/var/empty:/sbin/nologin
nixbld12:x:30012:30000:Nix build user 12:/var/empty:/sbin/nologin
nixbld13:x:30013:30000:Nix build user 13:/var/empty:/sbin/nologin
nixbld14:x:30014:30000:Nix build user 14:/var/empty:/sbin/nologin
nixbld15:x:30015:30000:Nix build user 15:/var/empty:/sbin/nologin
nixbld16:x:30016:30000:Nix build user 16:/var/empty:/sbin/nologin
nixbld17:x:30017:30000:Nix build user 17:/var/empty:/sbin/nologin
nixbld18:x:30018:30000:Nix build user 18:/var/empty:/sbin/nologin
nixbld19:x:30019:30000:Nix build user 19:/var/empty:/sbin/nologin
nixbld20:x:30020:30000:Nix build user 20:/var/empty:/sbin/nologin
nixbld21:x:30021:30000:Nix build user 21:/var/empty:/sbin/nologin
nixbld22:x:30022:30000:Nix build user 22:/var/empty:/sbin/nologin
nixbld23:x:30023:30000:Nix build user 23:/var/empty:/sbin/nologin
nixbld24:x:30024:30000:Nix build user 24:/var/empty:/sbin/nologin
nixbld25:x:30025:30000:Nix build user 25:/var/empty:/sbin/nologin
nixbld26:x:30026:30000:Nix build user 26:/var/empty:/sbin/nologin
nixbld27:x:30027:30000:Nix build user 27:/var/empty:/sbin/nologin
nixbld28:x:30028:30000:Nix build user 28:/var/empty:/sbin/nologin
nixbld29:x:30029:30000:Nix build user 29:/var/empty:/sbin/nologin
nixbld30:x:30030:30000:Nix build user 30:/var/empty:/sbin/nologin
nixbld31:x:30031:30000:Nix build user 31:/var/empty:/sbin/nologin
nixbld32:x:30032:30000:Nix build user 32:/var/empty:/sbin/nologin
~ # cat /etc/group
root:x:0:
nixbld:x:30000:nixbld1,nixbld2,nixbld3,nixbld4,nixbld5,nixbld6,nixbld7,nixbld8,nixbld9,nixbld10,nixbld11,nixbld12,nixbld13,nixbld14,nixbld15,nixbld16,nixbld17,nixbld18,nixbld19,nixbld20,nixbld21,nixbld22,nixbld23,nixbld24,nixbld25,nixbld26,nixbld27,nixbld28,nixbld29,nixbld30,nixbld31,nixbld32

Something jumps to your eyes?

tlaurion commented 1 year ago

@Hoverbear changed issue name to emphase on busybox usage use case, and to point that anything else expected from "embedded systems" might be cause of issue.

As stated before, I added user/group management commands to built busybox, added explicited kernel requirements per prior error resolution, but anything else non stated could be cause of error.

Everything properly extracted to tmp nix store, but error when attempting to move it under /nix.

Any insights here?

Working on heads branch to point to you so building a qemu target makes it replicable to you (will require host swtpm and similar build stack explicited to be present on host from apt call of circleci config in Heads repo, in branch).

Hoverbear commented 1 year ago

nix-installer always creates users on stable releases.

We have been experimenting with a branch that configures Nix to automagically allocate UIDs, it still requires groups though (and the ability to create these tempusers similar to systemd's dynamicUsers. That's only available on main right now. You're welcome to try, but I would be surprised if it solved all your issues (particularly the one you shared in https://github.com/DeterminateSystems/nix-installer/issues/446#issuecomment-1533737580.)

If you get digging into it, you might appreciate the configuration arguments of --logger pretty --log-directive "nix_installer=trace" to get a lot of information about what's happening.

I also pointed you to the single user install scripts (described in https://nixos.org/manual/nix/stable/installation/single-user.html). As you saw, root is not supported by that option. That makes sense: Nix doesn't want you to do builds as root. Perhaps you could find a way to workaround that.

tlaurion commented 1 year ago

Hey @Hoverbear sorry for the delay.

Modified my PoC script to look like:

user@heads-tests:~/heads$ cat initrd/bin/nixPoC
network-init-recovery
#rm wget busybox since we have u-root's version with /etc/ssl added in u-root.cpio per modules/u-root hacks
#with https://patch-diff.githubusercontent.com/raw/u-root/u-root/pull/2672.patch applied on top of u-root master
rm /bin/wget
wget https://install.determinate.systems/nix/tag/v0.8.0/nix-installer-x86_64-linux -O /tmp/nix-installer
chmod u+x /tmp/nix-installer
RUST_BACKTRACE=full /tmp/nix-installer install linux --extra-conf "sandbox = false" --logger pretty --log-directive "nix_installer=trace" --init none

Snippet of where it fails with more details:

  2023-05-11T20:59:28.176287Z DEBUG nix_installer::action::stateful: Completed: Place the Nix configuration in `/etc/nix/nix.conf`
    at src/action/stateful.rs:184
    in nix_installer::action::common::place_nix_configuration::place_nix_configuration
    in nix_installer::action::common::configure_nix::execute
    in nix_installer::action::stateful::try_execute
    in nix_installer::plan::install
    in nix_installer::cli::subcommand::install::execute
    in nix_installer::cli::execute

  2023-05-11T20:59:29.604763Z TRACE nix_installer: Executing
    at src/lib.rs:96
    in nix_installer::execute_command with command: HOME="/" NIX_SSL_CERT_FILE="/nix/store/ba4f8msp39cfvfpw3m7fsalb4psw347z-nss-cacert-3.83/etc/ssl/certs/ca-bundle.crt" "/nix/store/mc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3/bin/nix-env" "-i" "/nix/store/mc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3"
    in nix_installer::action::base::setup_default_profile::execute
    in nix_installer::action::base::setup_default_profile::setup_default_profile with unpacked_path: /nix/temp-install-dir
    in nix_installer::action::common::configure_nix::execute
    in nix_installer::action::stateful::try_execute
    in nix_installer::plan::install
    in nix_installer::cli::subcommand::install::execute
    in nix_installer::cli::execute

  2023-05-11T20:59:31.045818Z DEBUG nix_installer::diagnostics: Sending diagnostic to `https://install.determinate.systems/nix/diagnostic`
    at src/diagnostics.rs:174
    in nix_installer::diagnostics::send
    in nix_installer::plan::install
    in nix_installer::cli::subcommand::install::execute
    in nix_installer::cli::execute

  2023-05-11T20:59:31.808915Z ERROR nix_installer::cli::subcommand::install: 
   0: Install failure
   1: Error executing action
   2: Action `configure_nix` errored
   3: Action `setup_default_profile` errored
   4: Failed to execute command with status 1 `HOME="/" NIX_SSL_CERT_FILE="/nix/store/ba4f8msp39cfvfpw3m7fsalb4psw347z-nss-cacert-3.83/etc/ssl/certs/ca-bundle.crt" "/nix/store/mc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3/bin/nix-env" "-i" "/nix/store/mc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3"`, stdout: 
      stderr: installing 'nix-2.13.3'
      error: unable to load seccomp BPF program: Invalid argument
      (use '--show-trace' to show detailed location information)

   4: 

Location:
   /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/convert/mod.rs:726

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   0: nix_installer::cli::subcommand::install::execute
      at src/cli/subcommand/install.rs:63
   1: nix_installer::cli::execute
      at src/cli/mod.rs:37

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  <empty backtrace>

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.

Consider reporting this error using this URL: https://github.com/DeterminateSystems/nix-installer/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0AError%3A+%0A+++0%3A+Install+failure%0A+++1%3A+Error+executing+action%0A+++2%3A+Action+%60configure_nix%60+errored%0A+++3%3A+Action+%60setup_default_profile%60+errored%0A+++4%3A+Failed+to+execute+command+with+status+1+%60HOME%3D%22%2F%22+NIX_SSL_CERT_FILE%3D%22%2Fnix%2Fstore%2Fba4f8msp39cfvfpw3m7fsalb4psw347z-nss-cacert-3.83%2Fetc%2Fssl%2Fcerts%2Fca-bundle.crt%22+%22%2Fnix%2Fstore%2Fmc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3%2Fbin%2Fnix-env%22+%22-i%22+%22%2Fnix%2Fstore%2Fmc43d38fibi94pp5crfwacl5gbslccd0-nix-2.13.3%22%60%2C+stdout%3A+%0A++++++stderr%3A+installing+%27nix-2.13.3%27%0A++++++error%3A+unable+to+load+seccomp+BPF+program%3A+Invalid+argument%0A++++++%28use+%27--show-trace%27+to+show+detailed+location+information%29%0A%0A%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7C**version**%7C0.8.0%7C%0A%7C**os**%7Clinux%7C%0A%7C**arch**%7Cx86_64%7C%0A%0A%0A%23%23+SpanTrace%0A%0A%3Cdetails%3E%0A%0A%60%60%60%0ASpanTrace%3A%0A+++0%3A+nix_installer%3A%3Acli%3A%3Asubcommand%3A%3Ainstall%3A%3Aexecute%0A+++++++++++++at+src%2Fcli%2Fsubcommand%2Finstall.rs%3A63%0A+++1%3A+nix_installer%3A%3Acli%3A%3Aexecute%0A+++++++++++++at+src%2Fcli%2Fmod.rs%3A37%0A%60%60%60%0A%3C%2Fdetails%3E%0A%0A%23%23+Backtrace%0A%0A%3Cdetails%3E%0A%0A%60%60%60%0ABacktrace%3A%0A+++0%3A+%3Cunknown%3E%0A+++1%3A+%3Cunknown%3E%0A+++2%3A+%3Cunknown%3E%0A+++3%3A+%3Cunknown%3E%0A+++4%3A+%3Cunknown%3E%0A+++5%3A+%3Cunknown%3E%0A+++6%3A+%3Cunknown%3E%0A+++7%3A+%3Cunknown%3E%0A+++8%3A+%3Cunknown%3E%0A+++9%3A+%3Cunknown%3E%0A++10%3A+%3Cunknown%3E%0A++11%3A+%3Cunknown%3E%0A%0A%60%60%60%0A%3C%2Fdetails%3E
    at src/cli/subcommand/install.rs:205
    in nix_installer::cli::subcommand::install::execute
    in nix_installer::cli::execute

Installation failure, offering to revert...
Nix uninstall plan (v0.8.0)
tlaurion commented 1 year ago

So I get that bpf is missing here. Hmm. How to pack that under u-root I guess...

Wow. that's a lot of things to pack into firmware: https://packages.debian.org/bullseye/amd64/binutils-bpf/filelist

But you should definitely test for that bin presence prior of launching script prior of failing!

Hoverbear commented 1 year ago

Yeah, we probably should!

Hoverbear commented 1 year ago

Can you try with --extra-conf "sandbox = false"? That should remove most of the bpf requirements. (We do this in unprivileged user podman containers)

tlaurion commented 1 year ago

Can you try with --extra-conf "sandbox = false"?

chmod u+x /tmp/nix-installer RUST_BACKTRACE=full /tmp/nix-installer install linux --extra-conf "sandbox = false" --logger pretty --log-directive "nix_installer=trace" --init none

@Hoverbear already there on snippet above, which is why I'm a bit confused on the most basic requirements of this project (sandbox false should not use bpf? No?)

Throw my way other commands you want me try tagging me!

tlaurion commented 1 year ago

@Hoverbear already there on snippet above, which is why I'm a bit confused on the most basic requirements of this project (sandbox false should not use bpf? No?)

@Hoverbear some insights on kernel config options that seem required:

│This option optimizes the scheduler for common desktop workloads by │
│ automatically creating and populating task groups. This separation │
│ of workloads isolates aggressive CPU burners (like build jobs) from │
│ desktop applications. Task group autogeneration is currently based │
│ upon task session.

  • CONFIG_BPF_SYSCALL (seems required)
  • CONFIG_BPF_JIT (seems needed, and might fix the issue...)
Hoverbear commented 1 year ago

I'm not sure what the exact kernel requirements for Nix are, I'm sorry. It would be quite nice to be able to check as part of planning though. Perhaps I could find out...

tlaurion commented 1 year ago

As far as the error above:

error: unable to load seccomp BPF program: Invalid argument

This issue is similar to another one opened but on x86 and not in a container. BPF options were not compiled in kernel at time of posting the issue.

edolstra commented 1 year ago

Note that you can get around the BPF requirement by setting filter-syscalls = false in `nix.conf.