Cannot chroot to a different CPU architecture using `boot.binfmt.emulatedSystems`

chuangzhu commented 2 years ago

Describe the bug

Chroot to a guest system rootfs of a different CPU architecture fails. Reporting /bin/bash: No such file or directory.

Steps To Reproduce

Add boot.binfmt.emulatedSystems = [ "aarch64-linux" ] to the host's configuration and rebuild.

Fetch the tarball of a guest system for a different arch (Gentoo arm64 here):

[root@nixos-x86_64]# mkdir -p /mnt/gentoo-arm64 && cd /mnt/gentoo-arm64
[root@nixos-x86_64]# curl -LO https://bouncer.gentoo.org/fetch/root/all/releases/arm64/autobuilds/20220213T235221Z/stage3-arm64-systemd-20220213T235221Z.tar.xz

Extract the tarball and try to chroot into it:

[root@nixos-x86_64]# tar xf stage3-arm64-systemd-20220213T235221Z.tar.xz
[root@nixos-x86_64]# chroot /mnt/gentoo-arm64/ /bin/bash
chroot: failed to run command ‘/bin/bash’: No such file or directory
[root@nixos-x86_64]# systemd-nspawn -D /mnt/gentoo-arm64 -M gentoo-arm64
Spawning container gentoo-arm64 on /mnt/gentoo-arm64.
Press ^] three times within 1s to kill container.
execv(/bin/bash, /bin/sh) failed: No such file or directory
Container gentoo-arm64 failed with error code 1.

Expected behavior

Qemu correctly interprets /mnt/gentoo-amd64/bin/bash and execute it.

Additional context

Manually executing dynamic link loader on the bash executable under the guest rootfs works

[root@nixos-x86_64]# LD_LIBRARY_PATH=/mnt/gentoo-arm64/lib64 /mnt/gentoo-arm64/lib64/ld-linux-aarch64.so.1 /mnt/gentoo-arm64/bin/bash
bash-5.1#

Chroot to a guest rootfs of the same arch with the host works:

[root@nixos-x86_64]# mkdir -p /mnt/gentoo-amd64 && cd /mnt/gentoo-amd64
[root@nixos-x86_64]# curl -O https://bouncer.gentoo.org/fetch/root/all/releases/amd64/autobuilds/20220214T095322Z/stage3-amd64-systemd-20220214T095322Z.tar.xz
[root@nixos-x86_64]# tar xf stage3-amd64-systemd-20220214T095322Z.tar.xz 
[root@nixos-x86_64]# systemd-nspawn -D /mnt/gentoo-amd64/ -M gentoo-amd64
Spawning container gentoo-amd64 on /mnt/gentoo-amd64.
Press ^] three times within 1s to kill container.
gentoo-amd64 ~ #

Notify maintainers

@matthewbauer @shlevy @zhaofengli @polykernel @flokli

Metadata

[root@nixos-x86_64]# nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.99, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.6.0`
 - channels(root): `"nixos"`
 - channels(chuang): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

flokli commented 2 years ago

Looking around at https://unix.stackexchange.com/a/177122, it seems like chroot to non-native architectures only works if you copy the qemu-you-use-for-binfmt into your to-be-chrooted-into path, and all of its dependencies - which is why people usually use a static qemu executable for that.

I didn't check if nixpkgs can build qemu_full statically - but even if we can, you'd still need to manually copy/bind-mount that closure into the chroot.

chuangzhu commented 2 years ago

I didn't check if nixpkgs can build qemu_full statically - but even if we can, you'd still need to manually copy/bind-mount that closure into the chroot.

Manually copying isn't needed for other distributions as host, at least for Debian and Arch Linux I've tested.

root@debian:~# apt install qemu-user-static binfmt-support systemd-container
root@debian:~# mkdir /mnt/gentoo-arm64 && cd /mnt/gentoo-arm64
root@debian:/mnt/gentoo-arm64# wget https://bouncer.gentoo.org/fetch/root/all/releases/amd64/autobuilds/20220214T095322Z/stage3-amd64-systemd-20220214T095322Z.tar.xz
root@debian:/mnt/gentoo-arm64# tar xf stage3-arm64-systemd-20220213T235221Z.tar.xz
root@debian:/mnt/gentoo-arm64# systemd-nspawn -D /mnt/gentoo-arm64/ -M gentoo-arm64
Spawning container gentoo-arm64 on /mnt/gentoo-arm64.
Press ^] three times within 1s to kill container.
gentoo-arm64 ~ #

flokli commented 2 years ago

Hm, could this be some impure propagation of the dynamic linker of some sort?

I need to admit, I didn't dig deep enough into how this works…

zhaofengli commented 2 years ago

binfmt_misc has the F (fix binary) flag that preloads the emulator binary so it's always available even when the chroot doesn't have it, and it's what Debian does in their setup. However, this won't help in our current setup where we have a wrapper binary (or in our previous setup before #143060, a shell script).

We can look into patching our QEMU to recognize the P arguments directly and use static builds for binfmt.

zhaofengli commented 2 years ago

Pushed a WIP version in #160802 with caveats.

jcaesar commented 1 month ago

I'm working on this, for now only trying to get a statically linked version of qemu into nixpkgs in #314998.

My current workaround is to use a qemu packaged based on that PR, and then setting interpreter and wrapInterpreterInShell. (pkgs.pkgsStatic.qemu-user.override can be replaced with pkgs.pkgsStatic.callPackage …/qemu-user.nix.)

[Edit:] The exact options to set may require some consideration. see here

nixos-discourse commented 2 weeks ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/chroot-into-arm-container-with-systemd-nspawn/34735/7

NixOS / nixpkgs