bug: Fail to build root config when using remote builders on NixOS.

[voronind-com]

Are you following the right branch?

• [X] My Nixpkgs and Home Manager versions are in sync

Is there an existing issue for this?

• [X] I have searched the existing issues

Issue description

I have the exact same configuration for users and root. Normal users activate HM successfully, but the root user fails with this error:

> Starting Home Manager environment for root...
> Starting Home Manager activation
> Activating checkFilesChanged
> Activating checkLinkTargets
> Activating writeBoundary
> Activating createGpgHomedir
> Activating linkGeneration
> Cleaning up orphan links from /root
> No change so reusing latest profile generation 265
> Creating home file links in /root
> Activating createXdgUserDirectories
> Activating installPackages
> replacing old 'home-manager-path'
> installing 'home-manager-path'
> error: unable to execute 'ssh': No such file or directory
> cannot build on 'ssh-ng://nixbuilder': error: failed to start SSH connection to 'nixbuilder'
> Failed to find a machine for remote build!
> derivation: qjzvj1y2yqi7jarf7930djq7p9gp9p24-user-environment.drv
> required (system, features): (builtin, [])
> 1 available machines:
> (systems, maxjobs, supportedFeatures, mandatoryFeatures)
> ([x86_64-linux, i686-linux, aarch64-linux], 1, [benchmark, big-parallel, kvm, nixos-test], [])
> error: unable to start any build; remote machines may not have all required system features.
>        https://nixos.org/manual/nix/stable/advanced-topics/distributed-builds.html
> Oops, Nix failed to install your new Home Manager profile!
> Perhaps there is a conflict with a package that was installed using
> "nix-env -i"? Try running
>     nix-env -q
> and if there is a conflicting package you can remove it with
>     nix-env -e {package name}
> Then try activating your Home Manager configuration again.
> home-manager-root.service: Main process exited, code=exited, status=1/FAILURE
> home-manager-root.service: Failed with result 'exit-code'.
> Failed to start Home Manager environment for root.

I have openssh available system-wide and working.

I have remote builds enabled like this:

```nix { pkgs, lib, config, secret, ... }: with lib; let cfg = config.module.builder; serverKeyPath = "/root/.nixbuilder"; in { options = { module.builder = { server.enable = mkEnableOption "This is a builder server."; client.enable = mkEnableOption "This is a builder client."; }; }; config = mkMerge [ (mkIf cfg.server.enable { # Service that generates new key on boot if not present. # Don't forget to add new key to secret.ssh.buildKeys. systemd.services.generate-nix-cache-key = { wantedBy = [ "multi-user.target" ]; serviceConfig.Type = "oneshot"; path = [ pkgs.nix ]; script = '' [[ -f "${serverKeyPath}/private-key" ]] && exit mkdir ${serverKeyPath} || true nix-store --generate-binary-cache-key "nixbuilder-1" "${serverKeyPath}/private-key" "${serverKeyPath}/public-key" nix store sign --all -k "${serverKeyPath}/private-key" ''; }; # Add `nixbuilder` restricted user. users.groups.nixbuilder = {}; users.users.nixbuilder = { openssh.authorizedKeys.keys = secret.ssh.buildKeys; description = "Nix Remote Builder"; isNormalUser = true; createHome = lib.mkForce false; uid = 1234; home = "/"; group = "nixbuilder"; }; # Sign store automatically. # Sign existing store with: nix store sign --all -k /path/to/secret-key-file nix.settings = { trusted-users = [ "nixbuilder" ]; secret-key-files = [ "${serverKeyPath}/private-key" ]; }; }) (mkIf cfg.client.enable { # NOTE: Requires host private key to be present in secret.ssh.builderKeys. nix.buildMachines = [{ hostName = "nixbuilder"; protocol = "ssh-ng"; systems = [ "x86_64-linux" "i686-linux" "aarch64-linux" ]; maxJobs = 1; speedFactor = 2; mandatoryFeatures = [ ]; supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ]; }]; nix.distributedBuilds = true; nix.settings = let substituters = [ "ssh-ng://nixbuilder" ]; in { substituters = mkForce substituters; trusted-substituters = substituters; builders-use-substitutes = true; max-jobs = 0; trusted-public-keys = [ secret.ssh.builderKey ]; # require-sigs = false; # substitute = false; }; }) ]; } ```

Maintainer CC

No response

System information

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.9.6, NixOS, 24.11 (Vicuna), 24.11.20240624.2893f56`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `"nixos"`
 - nixpkgs: `/nix/store/bi5zxc0v5g6ylygdwyqzh280sccg3ykb-source`

[voronind-com]

Another relevant part is nix.conf:

# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
allowed-users = *
auto-optimise-store = true
builders-use-substitutes = true
cores = 0
experimental-features = nix-command  flakes
keep-derivations = true
keep-outputs = true
max-jobs = 0
min-free = 50000000000
require-sigs = true
sandbox = true
sandbox-fallback = false
substituters = ssh-ng://nixbuilder
system-features = nixos-test benchmark big-parallel kvm
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nixbuilder-1:Skghjixd8lPzNe2ZEgYLM9Pu/wF9wiZtZGsdm3bo9h0=
trusted-substituters = ssh-ng://nixbuilder
trusted-users = root
extra-sandbox-paths = 

If I remove the file the builder will be disabled and the switch happens correctly.

[voronind-com]

If I understand the issue correctly, HM needs ssh in its path when doing the switch because of a remote builder over ssh and jobs = 0.

[voronind-com]

I guess the issue is that ssh needed for the builder is missing in https://github.com/nix-community/home-manager/blob/d2493de5cd1da06b6a4c3e97f4e7d5dd791df457/modules/home-environment.nix#L680-L682

Sure I can add it to home.extraActivationPath, but shouldn't this be upstreamed for NixOS support?

Edit: can confirm that adding openssh to extraActivationPath fixes the issue.

[voronind-com]

@rycee I guess this is not needed for non-nixos installations as ssh client only needed in a scenario of using NixOS distributed builds. Can this be added to nixos module when nix.distributedBuilds is true only?