search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.89k stars 13.95k forks source link

nixos-rebuild switch fails when targeting aarch64 from x86 machines #177873

Open pyle opened 2 years ago

pyle commented 2 years ago

Describe the bug

G'day, c1bc8de9e3f174853df69acdc2fa594d34d5417c introduces a nixos-rebuild failure when --target-host/--build-host are set to a aarch64-linux host and you're running the command on an x86_64-linux machine.

Prior to 22.05 nixos-rebuild did not "reexec" when using flakes which means the script didn't try to nix build a derivation which has the system attribute as aarch64.

e.g. REXEC will atttempt to execute a derivation: $ nix --extra-experimental-features nix-command flakes build --out-link /tmp/nix-shell.gBYKHm/nixos-rebuild.ZFCrK5/nixos-rebuild .#nixosConfigurations."my-aarch64-system".config.system.build.nixos-rebuild --verbose --show-trace on an x86 platform causing the following error.

error: a 'aarch64-linux' with features {} is required to build '/nix/store/nmx9j676r0hf3f20c4jmhsvhnq8slwjk-nixos-rebuild.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

This will only trigger on $canRun commands like switch|boot|test

Steps To Reproduce

Steps to reproduce the behavior:

  1. Attempt to run nixos-rebuild --target-host aarch64-machine --build-host aarch64-machine --flake .#aarch64-machine from an x86 box
  2. Produces the above error.

Expected behavior

nixos-rebuild shouldn't attempt to reexec a systems flake if its architecture differs from its own OR nixos-rebuild should be built on the target machine :man_shrugging:

Workaround

You can bypass the reexec by adding the --fast flag.

Notify maintainers

@Profpatsch meta.maintainers @enzime commit author

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

Host machine:

 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.46, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.10.0pre20220610_45ebaab`
 - channels(pyle): `""`
 - channels(root): `"nixos-21.05.2362.b09c989b82f"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Target machine (since upgraded to 22.05 but previously on 21.11 until --fast workaround used:

 - system: `"aarch64-linux"`
 - host os: `Linux 5.10.115, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.8.1`
 - channels(root): `"nixos-21.05.2501.94c989365d5"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

If you need any more details just let me know :)

nixos-discourse commented 1 year ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/advice-which-lean-nixos-server-devops-tool-to-use/23778/8

merspieler commented 6 months ago

I've got the same problem but --fast doesn't fix it. Any idea when this will be fixed?

noisersup commented 6 months ago

Same issue here, is there any reason it is not fixed yet? (not rushing anyone, just making sure it's a bug that can be worked on)

adamcstephens commented 6 months ago

Another workaround could be adding the aarch64 host as a remote builder on the source host.

Or you could try using binfmt stubs to emulate, but if you’re targeting a remote host anyway then it’s probably better to have that host do the building.

roberth commented 3 weeks ago

Another workaround could be adding the aarch64 host as a remote builder on the source host.

This will produce a nixos-rebuild script whose shebang points to an aarch64 bash, which can't run natively on the (e.g. x86_64-linux) source host. Not at all if it's a macOS source host.

Potential solutions: