nixos-rebuild --target-host will inherit system from invocation and not target host (flake)

[pyle]

Describe the bug

x86_64 packages are built and attempted to be installed when targeting an aarch64-linux machine.

Example: nixos-rebuild build --target-host r2 --flake .#r2 Error:

error: a 'x86_64-linux' with features {} is required to build '/nix/store/b1kxbl7araqsqhfbd5m77vf41kiab6gh-append-initrd-secrets.drv', but I am a 'aarch64-linux' with features {benchmark, big-parallel, gccarch-armv8-a, kvm, nixos-test}

I'm new to NixOS, but im guessing that because we're grabbing architecture from the local system this is why this behaviour is happening?

ex. nixSystem() is defined via local invocation of uname -a and then used a few lines later when defining nixStorePath

Steps To Reproduce

Steps to reproduce the behavior:

1. Have nixosConfiguration that can be built on aarch64 e.g. I've copied this flake to the device and can run nixos-rebuild switch --flake '.#' without any issue regarding architecture.
2. nixos-rebuild build --target-host <my_aarch64_target_host> --flake .#targetHostNameThatIsAarch64
3. Observe error about the packages requiring x86 but being aarch64 (above).

Expected behavior

nixos-rebuild builds/switches using the correct architecture of the target-host.

Additional context

From what I can work out we can't pass args like --argstr system aarch64-linux to nixos-rebuild as we're likely assuming same architecture.

Notify maintainers

@chvp

Metadata

Machine invoking nixos-rebuild --target-host

 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.48, NixOS, 21.05.20210710.9376bf7 (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210601_5985b8b`
 - channels(pyle): `""`
 - channels(root): `"nixos-21.05.2362.b09c989b82f"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos

--target-host in question

 - system: `"aarch64-linux"`
 - host os: `Linux 5.10.48, NixOS, 21.05.20210710.9376bf7 (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210601_5985b8b`
 - channels(root): `"nixos-21.05.2501.94c989365d5"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module: nixos-rebuild

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/building-a-flake-based-nixos-system-remotely/11309/8

[chvp]

Are you sure that your shell isn't stripping #targetHostNameThatIsAarch64 when executing nixos-rebuild build --target-host <my_aarch64_target_host> --flake .#targetHostNameThatIsAarch64? # is usually a comment character.

[pyle]

That's a good point, but even so building with --flake '.#targetHostNameThatIsAarch64' still is still building the same package and produces the same error:

error: a 'x86_64-linux' with features {} is required to build '/nix/store/b1kxbl7araqsqhfbd5m77vf41kiab6gh-append-initrd-secrets.drv', but I am a 'aarch64-linux' with features {benchmark, big-parallel, gccarch-armv8-a, kvm, nixos-test}

My flake at the moment has definitions for both architectures e.g.

flake-utils.lib.eachSystem [ "x86_64-linux" "aarch64-linux" ]  (
...
packages.nixosConfigurations = {
  r2 = nixpkgs.lib.nixosSystem {};
};
...

If i remove my x86 option, so the package only has aarch64 to build, I recieve the following:

error: flake 'git+file:///persist/pyle/.dotfiles' does not provide attribute 'packages.x86_64-linux.nixosConfigurations."r2".config.system.build.toplevel.drvPath', 'legacyPackages.x86_64-linux.nixosConfigurations."r2".config.system.build.toplevel.drvPath' or 'nixosConfigurations."r2".config.system.build.toplevel.drvPath'

[chvp]

Ah, your nixosConfigurations should not be under packages, but a top-level attribute. nixosConfigurations declare their system by passing system = "<arch>-linux"; to nixpkgs.lib.nixosSystem.

[pyle]

@chvp that fixed it! Had these under packages to ~fix~ mask another issue but after some re-adjusting it works as intended. Closing issue. Thx!

[mawalu]

@pyle could you show the config that worked in the end? I'm currently experiencing the same issue even though nixosConfiugrations is a top level output in my flake:

Repro example

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.05";

  outputs = { self, nixpkgs }:
  {
    nixosConfigurations.golf = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [ ./golf/configuration.nix ];
    };
  };
}

$ nixos-rebuild --build-host root@remote --target-host root@remote --flake .#golf switch
error: a 'aarch64-linux' with features {} is required to build '/nix/store/i187zblgxcav4dblgir1xnryrp2zy4ky-nixos-rebuild.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

[pyle]

@mawalu that config looks correct to me. You may be running into another issue i found when deploying to aarch64 on x86 hosts https://github.com/NixOS/nixpkgs/issues/177873.

Workaround is to add --fast to your nixos-rebuild command to bypass some flake smarts.

[mawalu]

Thanks, yes that fixes the problem. I think I tried --no-build-nix, which does the same think as --fast according to the docs, but does not solve this issue.