nixos-rebuild fails cross-build a flake

wentasah commented 2 years ago

Describe the bug

nixos-rebuild does not work when when building a flake and the host and target systems are not the same.

The problem was originally posted by @chvp in https://github.com/NixOS/nixpkgs/issues/153981#issuecomment-1083128534.

Steps To Reproduce

On a Darwin system run: darwin# nixos-rebuild switch --flake ... --build-host linuxhost --target-host linuxhost --config ...
Observe this error: error: a 'x86_64-linux' with features {} is required to build '/nix/store/yq1xkqyhb6qh9z4j1bw7b9kdfb0qai2z-nixos-rebuild.drv', but I am a 'aarch64-darwin' with features {benchmark, big-parallel, nixos-test}

(@chvp, please correct the above if it's not what you are doing. I don't have a darwin system to test that.)

Expected behavior

No error is produced as was the case before #153981.

Additional context

Recently, nixos-rebuild gained a feature to reexecute a new version of itself, before building a flake system. This is the same thing that's being done for non-flake systems. The TODO comment requesting addition of the same thing for flakes was added here.

One way of resolving the issue could be to disable reexecution if the host and target systems are different. However, I'm not sure whether it's a good thing to have different behavior between cross and native builds. I think that it's generally not possible to build a host version of nixos-rebuild to reexecute itself, because the flake being built for the target host might not have any configuration for the host system, just a configuration for the target. Maybe, I'm missing something. Any suggestions?

Notify maintainers

@Profpatsch (@Enzime @Infinisil)

roberth commented 2 years ago

You do mean cross, not remote right? Otherwise I typed the following in vain:

To cross-build a configuration, you have to set the cross-build attributes, so that the derivation system field is set to a correct value for your --build-host.

Inferring the cross configuration from your --build-host flies in the face of reproducibility, as you could accidentally cross-compile by specifying a different host. Equally, it can not even always be inferred, because some hosts can build natively for multiple systems.

See the nixpkgs.* options https://search.nixos.org/options?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=nixpkgs+system

If you use nixpkgs.pkgs = nixpkgs.legacyPackages.foo-linux, which might take advantage of inter-flake eval caching at some point, you need something like https://github.com/NixOS/nixpkgs/pull/160061.

wentasah commented 2 years ago

Thanks for the comment. I meant cross and remote, so you comment applies. I understand that changing the system attribute should not be done based on the --build-host (or any other) switch, that's clear. Cross-building the nixos-rebuild script would require either changing the flake or requiring the flake to have parameters input as in #160061. I think that this would be too much complexity for what I think is currently a niche use case. The solution in #166502 is probably sufficient. At least for now.

RyanGibb commented 1 year ago

I'm also experiencing this issue building for AArch64 on x86-64 https://github.com/NixOS/nixpkgs/issues/200398.

I can't get a nix build with a remote builder to work either:

nix build -L /etc/nixos#nixosConfigurations.rasp-pi.config.system.build.toplevel --builders 'ssh://rasp-pi aarch64-linux'
error: a 'aarch64-linux' with features {} is required to build '/nix/store/vxqx0agjchd2il48p1fip5hq7apb1vhq-NetworkManager.conf.drv', but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

https://github.com/NixOS/nixpkgs/issues/60101 seems relevant as it pertains to remote cross compilation.

RyanGibb commented 1 year ago

I was missing --max-jobs 0 [0].

A workaround is [1]:

$ sudo nix build /etc/nixos#nixosConfigurations.rasp-pi.config.system.build.toplevel --builders 'ssh-ng://rasp-pi aarch64-linux' --max-jobs 0 builders-use-substitutes --option builders-use-substitutes true
$ ssh root@rasp-pi nix-env -p /nix/var/nix/profiles/system --set $(readlink ./result)
$ ssh root@rasp-pi /nix/var/nix/profiles/system/bin/switch-to-configuration switch

[0] https://github.com/NixOS/nixpkgs/issues/200398#issuecomment-1318852022 [1] https://github.com/NixOS/nixpkgs/issues/200398#issuecomment-1318899763