binfmt armv7 not working on aarch64 linux on Apple M1 CPU

[patryk4815]

Describe the bug

binfmt not working on aarch64 linux on Apple M1 CPU

Apple M1 CPU does not support aarch32/armv7 architecture: https://news.ycombinator.com/item?id=27278019

[root@nixos:~]# cat /run/binfmt/armv7l-linux
#!/nix/store/ldyxgarwyhqfvsck38g0s8caxqgmxcfk-bash-5.1-p16/bin/sh
exec -- /nix/store/ldyxgarwyhqfvsck38g0s8caxqgmxcfk-bash-5.1-p16/bin/bash -c '"$@"' -- "$@"

In /run/binfmt/armv7l-linux should be qemu-arm not just bash

Steps To Reproduce

Steps to reproduce the behavior:

1. nix-build '<nixpkgs>' -A pkgsCross.armv7l-hf-multiplatform.hello

2. 
   [root@nixos:~]# ./result/bin/hello
   -bash: ./result/bin/hello: cannot execute binary file: Exec format error

[root@nixos:~]# file ./result/bin/hello ./result/bin/hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/s07i6932bbg7q9ddis3k6rx1i52jqjyh-glibc-armv7l-unknown-linux-gnueabihf-2.34-115/lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, not stripped

[root@nixos:~]# qemu-arm ./result/bin/hello Hello, world!

3. Add `boot.binfmt.emulatedSystems = [ "armv7l-linux" ];` into `/etc/nixos/configuration.nix`
4. `nixos-rebuild switch`
5. `./result/bin/hello` <- it cause infinite loop of `bash -> bash -> bash -> ...`
```console
[root@nixos:~]# strace -p 297658 -e execve
strace: Process 297658 attached
execve("/nix/store/ldyxgarwyhqfvsck38g0s8caxqgmxcfk-bash-5.1-p16/bin/bash", ["/nix/store/ldyxgarwyhqfvsck38g0s"..., "-c", "\"$@\"", "--", "./result/bin/hello"], 0x3f8a4210 /* 36 vars */) = 0
execve("./result/bin/hello", ["./result/bin/hello"], 0x14d7c210 /* 36 vars */) = 0
execve("/nix/store/ldyxgarwyhqfvsck38g0s8caxqgmxcfk-bash-5.1-p16/bin/bash", ["/nix/store/ldyxgarwyhqfvsck38g0s"..., "-c", "\"$@\"", "--", "./result/bin/hello"], 0xbdc3210 /* 36 vars */) = 0
execve("./result/bin/hello", ["./result/bin/hello"], 0x2e33e210 /* 36 vars */) = 0
execve("/nix/store/ldyxgarwyhqfvsck38g0s8caxqgmxcfk-bash-5.1-p16/bin/bash", ["/nix/store/ldyxgarwyhqfvsck38g0s"..., "-c", "\"$@\"", "--", "./result/bin/hello"], 0x1041d210 /* 36 vars */) = 0
execve("./result/bin/hello", ["./result/bin/hello"], 0x3543c210 /* 36 vars */) = 0
execve("/nix/store/ldyxgarwyhqfvsck38g0s8caxqgmxcfk-bash-5.1-p16/bin/bash", ["/nix/store/ldyxgarwyhqfvsck38g0s"..., "-c", "\"$@\"", "--", "./result/bin/hello"], 0x10f52210 /* 36 vars */) = 0
execve("./result/bin/hello", ["./result/bin/hello"], 0xe030210 /* 36 vars */) = 0
execve("/nix/store/ldyxgarwyhqfvsck38g0s8caxqgmxcfk-bash-5.1-p16/bin/bash", ["/nix/store/ldyxgarwyhqfvsck38g0s"..., "-c", "\"$@\"", "--", "./result/bin/hello"], 0x1599b210 /* 36 vars */) = 0
execve("./result/bin/hello", ["./result/bin/hello"], 0x24de4210 /* 36 vars */) = 0
execve("/nix/store/ldyxgarwyhqfvsck38g0s8caxqgmxcfk-bash-5.1-p16/bin/bash", ["/nix/store/ldyxgarwyhqfvsck38g0s"..., "-c", "\"$@\"", "--", "./result/bin/hello"], 0x10719210 /* 36 vars */) = 0

Expected behavior

Possible to run armv7 binary using binfmt. Now it is not possible.

Screenshots

Notify maintainers

@matthewbauer

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[root@nixos:~]# nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 5.17.3, NixOS, 22.05 (Quokka), 22.05.20220425.6a32390`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.8.0`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[Lucus16]

source of the problem is here: https://github.com/NixOS/nixpkgs/blob/83beeb19acf9e39cfd941f5514dc55a3e63f4175/lib/systems/parse.nix#L189

[patryk4815]

Maybe we should remove this "magic" choice? And if the user specifies that he wants e.g. boot.binfmt.emulatedSystems = ["armv7l-linux" "aarch64-linux"]; will get qemu-arm and qemu-aarch64.

[Majiir]

@patryk4815 Are you still having this issue?

On my x86_64-linux machine, /run/binfmt/armv7l-linux is a symlink to a binary and not a bash script as you posted above. However, I still cannot run armv7l binaries directly:

$ nix build nixpkgs#pkgsCross.armv7l-hf-multiplatform.hello

$ ./result/bin/hello 
bash: ./result/bin/hello: cannot execute binary file: Exec format error

$ qemu-arm ./result/bin/hello
Hello, world!

whereas I can with aarch64:

$ nix build nixpkgs#pkgsCross.aarch64-multiplatform.hello

$ ./result/bin/hello
Hello, world!

$ qemu-aarch64 ./result/bin/hello
Hello, world!

[Majiir]

Upon further testing, it looks like my issue is fixed by restarting systemd-binfmt.service. I had added armv7l-linux to boot.binfmt.emulatedSystems and then ran nixos-rebuild test, but it didn't fully take effect until restarting the service.

[patryk4815]

@Majiir issue still exists on 64bit only aarch CPU (without 32bit support). Nixpkgs is incorrectly detecting architecture for arm32. He thinks it is possible to run arm32 program on aarch64 cpu, but is not supported on Apple M1 cpu.

[dguibert]

I had the same issue on a x86_64 system, restarting systemd-binfmt.service fixes the issue. armv7-linux was already into boot.binfmt.emulatedSystems in the previous system generation.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/exec-format-error-when-binfmt-building/43038/4