nix-build does not honor NIX_SSHOPTS when building system derivation

[wamserma]

If the nix store is shared via (legacy) ssh, nix-buildcalled from the shell works just fine, e.g. copies closures from the host given via extra-binary-caches. But when called from nixos-rebuildif fails to authenticate with the private key and tries Password-Authentication.

Example: On the server, run

nix-build '<nixpkgs>' --no-out-link -A chocolateDoom 

On the client run

NIX_SSHOPTS='-i /root/.ssh/id_nix_store'
nix-build '<nixpkgs>' --no-out-link -A chocolateDoom --option extra-binary-caches ssh://nix-ssh@server

Paths are copied as expected from the server, but when nix-build is called here: https://github.com/NixOS/nixpkgs/blob/82875a20ba444110396f95537b18247898e40e22/nixos/modules/installer/tools/nixos-rebuild.sh#L147 the call fails and I get (with tracing enabled, hence the one-off in the line numbers)

++(/nix/store/ci67b4iqvypfjw5f6vgwc7qssabcg08w-nixos-rebuild/bin/nixos-rebuild:148): nixBuild(): nix-build '<nixpkgs/nixos>' --no-out-link -A system -I nixpkgs=/root/nixpkgs --option extra-binary-caches ssh://nix-ssh@server
error (ignored): cannot connect to 'nix-ssh@server'
error: cannot connect to 'nix-ssh@server'

If I run (dunno if that makes sense at all):

NIX_SSHOPTS='-i /root/.ssh/id_nix_store'
nix-build '<nixpkgs/nixos>' --no-out-link -A system -I nixpkgs=/root/nixpkgs --option extra-binary-caches ssh://nix-ssh@server

i get for example

warning: substituter 'ssh://nix-ssh@server' does not have a valid signature for path '/nix/store/cgqaapw3899c6n70aqpnyclxj0s5ldr9-udev-rules'
writing to file: Broken pipe

whenever a closure is to be copied from server.

Something similar happened to @tbenst, reported at https://github.com/NixOS/nixpkgs/issues/76860, but somehow that resolved by dark magic while my problem persists.

[wamserma]

Update: The broken pipe was due to the amount of derivations requested and the pipe to the server timing out. Adding -o ServerAliveInterval=15 -o ServerAliveCountMax=3000 to NIX_SSHOPTSallowed me to build the system derivation (e.g. pull all the Gigabytes from the store served via SSH).

What remains is the NIX_SSHOPT ignored when invocation happens through nixos-rebuild.sh. Using ssh-agentalso doesn't make a difference, on the server side I always see a try for interactive authentication.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[stale[bot]]

I closed this issue due to inactivity. → More info