Question: nix-store: command not found

[lontivero]

My deploys fail with nix-store: command not found and I don't know how to find the problem. How can I find the source of this error? Clearly it is not deploy-rs but some bash script.

$ deploy --ssh-opts="-p 22220 -v"
🚀 ℹ️  [deploy] [INFO] Running checks for flake in .
warning: Git tree '/home/user/Projects/flakes/systemd' is dirty
trace: warning: system.stateVersion is not set, defaulting to 24.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
trace: warning: system.stateVersion is not set, defaulting to 24.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
warning: unknown flake output 'deploy'
🚀 ℹ️  [deploy] [INFO] Evaluating flake in .
warning: Git tree '/home/user/Projects/flakes/systemd' is dirty
trace: deploy-rs#lib.setActivate is deprecated, use activate.noop, activate.nixos or activate.custom instead
trace: warning: system.stateVersion is not set, defaulting to 24.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
🚀 ℹ️  [deploy] [INFO] The following profiles are going to be deployed:
[production.bitcoin]
user = "vmuser"
ssh_user = "vmuser"
path = "/nix/store/gm52zyy818mxqz3s9jdwf5plv9pac77h-activatable-activate"
hostname = "localhost"
ssh_opts = ["-p", "22220", "-v"]

🚀 ℹ️  [deploy] [INFO] Building profile `bitcoin` for node `production`
🚀 ℹ️  [deploy] [INFO] Copying profile `bitcoin` to node `production`
OpenSSH_9.7p1, OpenSSL 3.0.14 4 Jun 2024
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 17: Applying options for *
<sip>
debug1: Next authentication method: password
vmuser@localhost's password:
Authenticated to localhost ([127.0.0.1]:22220) using "password".
<sip>
debug1: Sending command: nix-store --serve --write
bash: line 1: nix-store: command not found
debug1: client_input_channel_req: channel 1 rtype exit-status reply 0
debug1: client_input_channel_req: channel 1 rtype eow@openssh.com reply 0
debug1: channel 1: free: client-session, nchannels 2
error: cannot connect to 'vmuser@localhost'
🚀 ❌ [deploy] [ERROR] Failed to push profile to node production: production

In the VM auth logs I can verify it connected successfully and then disconnected. The thing that I don't understand is the:

debug1: Sending command: nix-store --serve --write bash: line 1: nix-store: command not found

what is running that command and how is it that nix-store is not found when host runs nixos and guest has nix (multiuser) installed.

Any clue?

Flake

``` { inputs = { systemd-nix = { url = github:serokell/systemd-nix; inputs.nixpkgs.follows = "nixpkgs"; # Make sure the nixpkgs version matches }; deploy-rs.url = "github:serokell/deploy-rs"; wasabi.url = "github:walletwasabi/walletwasabi"; }; outputs = { self, nixpkgs, wasabi, deploy-rs, systemd-nix }: let pkgs = nixpkgs.legacyPackages.x86_64-linux; wasabi-backend = wasabi.packages.x86_64-linux.default; wasabi-service = systemd-nix.lib.x86_64-linux.mkUserService "wasabi" { description = "Runs Wasabi backend service"; path = [ ]; serviceConfig = { Type = "oneshot"; ExecStart = "${wasabi-backend}"; }; }; bitcoind = pkgs.bitcoind-knots; bitcoind-config = pkgs.writeText "bitcoin.conf" '' daemon=1 softwareexpiry=0 whitebind=127.0.0.1:8333 corepolicy=1 mempoolreplacement=fee,otpin ''; bitcoind-service = systemd-nix.lib.x86_64-linux.mkUserService "bitcoin" { description = "Bitcoin daemon"; path = [ ]; serviceConfig = { Type = "oneshot"; ExecStart = "${bitcoind} -conf=${bitcoind-config}"; After = ["network-online.target"]; Wants = ["network-online.target"]; }; }; in { packages.x86_64-linux = { wasabi-service = wasabi-service; bitcoind-service = bitcoind-service; }; deploy = { nodes.production = { hostname = "localhost"; sshOpts = [ "-p" "22220" ]; sshUser = "vmuser"; profiles.bitcoin = { user = "bitcoin-user"; path = deploy-rs.lib.x86_64-linux.setActivate bitcoind-service "activate"; }; }; }; }; } ```

Update: if I dd the --build-remote switch the error message changes to:

debug1: Sending command: nix-daemon --stdio bash: line 1: nix-daemon: command not found

[lontivero]

[lontivero]

The problem was nix binaries were not in the PATH when accessing through ssh. The solution was to add the missing paths to the /etc/environment file in my Ubuntu 22.04 guest.