unshare: unshare failed: Invalid argument

[tbaumann]

› nix log  '/nix/store/9rsiimwgjikqbl7kscqylpqwl1m9siz7-btrfs-fs.img.drv^*'
++ mkdir -p ./files
++ mkdir ./files/boot
++ /nix/store/x8dl4qqc8nb07bggpi03fb5zjjshdyjj-extlinux-conf-builder.sh -g 20 -t 5 -c /nix/store/zip59r44izangazz7npcbj5nzddp245q-nixos-system-nixos-24.05.20240914.8f7492c -d ./files/boot
/nix/store/x8dl4qqc8nb07bggpi03fb5zjjshdyjj-extlinux-conf-builder.sh: line 138: cd: /nix/var/nix/profiles: No such file or directory
++ mkdir -p ./rootImage/nix/store
++ xargs -I % cp -a --reflink=auto % -t ./rootImage/nix/store/
++ GLOBIGNORE=.:..
++ shopt -u dotglob
++ for f in ./files/*
++ cp -a --reflink=auto -t ./rootImage/ ./files/boot
++ cp /nix/store/xz2541lvyr60ap14q6a2v63lz7y3gwss-closure-info/registration ./rootImage/nix/nix-path-registration
++ mv ./rootImage rootSubVol
++ mkdir ./rootImage
++ mv ./rootSubVol ./rootImage//@
++ '[' -d ./rootImage//@//boot ']'
++ mv ./rootImage//@//boot ./rootImage//@boot
++ '[' -d ./rootImage//@//nix ']'
++ mv ./rootImage//@//nix ./rootImage//@nix
++ touch /nix/store/znbmnhcpd9rsid6a62h920s5ss55nlnc-btrfs-fs.img
++ faketime -f '1970-01-01 00:00:01' unshare -U -r mkfs.btrfs -L root -U 18db6211-ac36-42c1-a22f-5e15e1486e0d --subvol /@ --subvol /@boot --subvol /@nix -r ./rootImage --shrink /nix/store/znbmnhcpd9rsid6a62h920s5ss55nlnc-btrfs-fs.img
unshare: unshare failed: Invalid argument
+ exitHandler
+ exitCode=1
+ set +e
+ '[' -n '' ']'
+ ((  1 != 0  ))
+ runHook failureHook
+ local hookName=failureHook
+ shift
+ local 'hooksSlice=failureHooks[@]'
+ local hook
+ for hook in "_callImplicitHook 0 $hookName" ${!hooksSlice+"${!hooksSlice}"}
+ _eval '_callImplicitHook 0 failureHook'
+ declare -F '_callImplicitHook 0 failureHook'
+ eval '_callImplicitHook 0 failureHook'
++ _callImplicitHook 0 failureHook
++ local def=0
++ local hookName=failureHook
++ declare -F failureHook
++ type -p failureHook
++ '[' -n '' ']'
++ return 0
+ return 0
+ '[' -n '' ']'
+ return 1

I don't see anything obviously wrong with the unshare arguments. It's probably specific to my system.

› nix shell nixpkgs#libfaketime.out -c faketime  -f '1970-01-01 00:00:01' unshare -U -r echo foo
foo

[tbaumann]

Building without obviously works. But it would be better if it could just be made to work.

[nakato]

unshare: unshare failed: Invalid argument

Oh, that's a shame.

While it doesn't change the fact that unshare will need to go away, I am curious about the following.

What distro you're on? Does it have CONFIG_USER_NS in /proc/config.gz? Does it have the following file and what is it set to cat /proc/sys/user/max_user_namespaces? Apparently Debian and a few others have custom sysctl knobs that disable userns's to regular users as well. Does unshare -U -- id -u work at your shell? Does unshare -U -r -- id -u work at your shell? Anything odd about your environment worth mentioning, such as nix running in a rootless container/a container/etc.

---

Now, on to how to fix this. There are a couple of options, but a revert is not on the table with this one sadly. Prior to the merge-commit 987002c, which began using mkfs.btrfs with subvol support and unshare, the btrfs-subvol image was subtly broken, and the plain btrfs image was fully broken. ~I presume this is actually a regression, but I'm not sure when it came in, and it could be due to kernel, compiler, glibc, or mkfs.btrfs. I rolled back btrfs-progs to the same version of btrfs-progs in nixpkgs, and the issues remained, so it's not caused by btrfs-progs v6.11.~ ~Regression occurs between btrfs-progs v6.9.2 and v6.10.1.~ Bisected to https://github.com/kdave/btrfs-progs/commit/c6464d3f99ed1dabceff1168eabb207492c37624 between v6.10 and v6.10.1

The switch to unshare was due to the fakeroot command not working, and all the files created in the btrfs image were owned by UID 1000 and GID 100. This means all contents of /nix/store, /boot and the directories /, /nix, /nix/store, /boot were owned in this manner. The scripts that previously setup subvols on first-boot masked the worst of this, the ownership of /, which prevents sshd from starting as well as systemd-tmpfiles, which then breaks other services (dhcpcd, etc) as well due to missing directories. While going back to the RAMIFY scripts would allow it to build again, that opens up the system to being modified by UID 1000(First user), and some files GID 100 (users group by default), and that's a security problem, which is why revert is off the table.

1. chown on first-boot

Probably the simplest, on first-boot everything should be owned by root, so we could chown everything to root on first-boot. It would wreak havoc if it ever re-ran, and it would cause issues with user-pre-populated files.

2. Fix libfakeroot

Determine why fakeroot is failing (missing a function hook?) and fix it.

3. Patch mkfs.btrfs

Figuire out where mkfs.btrfs gets its ownership from, and hard-code it to root.

[nakato]

For now, I'm taking option 3. I've got a patch that will simply make everything be owned by root.

[tbaumann]

What distro you're on? Does it have CONFIG_USER_NS in /proc/config.gz?

shouldn't matter. It's all from stdenv and nativeBuildInputs = [ ... util-linux ]. Anyway, My os is Nixos current stable.

Does it have the following file and what is it set to cat /proc/sys/user/max_user_namespaces

› cat /proc/sys/user/max_user_namespaces
191895

Does unshare -U -- id -u work at your shell? Does unshare -U -r -- id -u work at your shell?

~ 
› unshare -U -- id -u
65534

~ 
› unshare -U -r -- id
uid=0(root) gid=0(root) groups=0(root),65534(nogroup)

In my shell even the entire mkfs command works like in the source. Just not in the build env,

Anything odd about your environment worth mentioning, such as nix running in a rootless container/a container/etc.

Not that I can think of. All pretty standard. https://github.com/tbaumann/nix-conf/blob/main/nix/common/core.nix#L17

It's got to be a side effect for sure, because it must have worked for you. But I really don't see how. There isn't a debug mode. Maybe I should try strace to see where it fails.

Maybe fakeroot was broken the same way faketime was too. faketime -f "1970-01-01 00:00:01" ls -l does not give all files fake [acm]time. But as far as I understood it should. only the date command is bent around correctly.

But I see no reason why LD_PRELOAD should be broken on Nix. So I'm probably just holding it wrong or have wrong expectations.

Maybe I can play around more tomorrow...

[nakato]

Does it have CONFIG_USER_NS in /proc/config.gz?

shouldn't matter. It's all from stdenv and nativeBuildInputs = [ ... util-linux ].

User namespaces are a kernel feature, so it's an impurity. But as you can use it as your user, then it's in your kernel.

I wonder if it is removed through syscall filtering done by the nix-daemon. Syscall filtering is big lists of allow/deny, and architecture does come into play with that, so that's possibly the cause of the difference.

Anyway, My os is Nixos current stable.

That's quite literally the best case scenario, no odd distro-patches coming into play here.

It's got to be a side effect for sure, because it must have worked for you.

Yea, it's odd, the environment I'm building in has very little customisation done to it, and has no virtualisation/containerisation configuration included. It's a orangepi5b, using this repo to configure it, with a nixpkgs kernel, plus impermanence and some network configuration.

This has got to be an x86_64-linux vs aarch64-linux impurity that is coming in from somewhere.

But I see no reason why LD_PRELOAD should be broken on Nix

Yea, in producing a quick hack to remove unshare usage, I bisected btrfs-progs, and stat is now no-longer happening in the btrfs binary, it happens when mkfs.btrfs calls a libc function, nftw, which handles the stat and passes the stat struct back to mkfs.btrfs. That means libc is calling stat within itself, LD_PRELOAD doesn't work.

Anyways, I've created a workaround, removing unshare, and I've merged it to HEAD. e27fb7e96f3faf0e4b34dc34cc7eefde1e47b40d

[nakato]

Odd, I can run this on my x86_64 laptop.

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = {
    self,
    nixpkgs,
    ...
  }: let
    pkgs = nixpkgs.legacyPackages.x86_64-linux;
  in {
    packages.x86_64-linux = {
      test = pkgs.stdenv.mkDerivation {
        name = "unshare test";
        nativeBuildInputs = [ pkgs.util-linux ];
        buildCommand = ''
          mkdir $out
          id > $out/whoami.builduser
          unshare -U -r id > $out/whoami.unshare
        '';
      };
    };
  };
}

grep '' result/*
result/whoami.builduser:uid=1000(nixbld) gid=100(nixbld) groups=100(nixbld)
result/whoami.unshare:uid=0(root) gid=0(root) groups=0(root)

[nakato]

I'm going to track the upstream bugs in #21

[tbaumann]

really odd

result/whoami.builduser:uid=1000(nixbld) gid=100(nixbld) groups=100(nixbld)
result/whoami.unshare:uid=0(root) gid=0(root) groups=0(root)