Setting PATH in NIX_SSHOPTS no longer works!

[Fuuzetsu]

Describe the bug

Due to https://github.com/NixOS/nix/issues/1078 , we have been using nix-copy-closure (or nix copy, whichever) with

NIX_SSHOPTS='PATH=/nix/var/nix/profiles/default/bin:"$PATH"'

This worked! However upon updating nix to 2.15, this stopped working for me. I made a shim in /tmp/bin to show the ssh command:

[shana@aya:~/programming/engine]$ cat /tmp/bin/ssh 
#!/usr/bin/env bash
arr=( "$@" )
echo ssh "$@" > /tmp/ran
for i in "${arr[@]}"; do
    echo "$i" >> /tmp/ran
done

/run/current-system/sw/bin/ssh "$@"

and ran with known-working nix 2.14:

[shana@aya:~/programming/engine]$ PATH=/tmp/bin:"$PATH" NIX_SSHOPTS='PATH=/nix/var/nix/profiles/default/bin:"$PATH"' $(nix-build --no-out-link ~/programming/nixpkgs -A nixVersions.nix_2_14)/bin/nix copy --from ssh://kobuta.production.tsuru.capital /nix/store/ch8jsr2qdymmz0xqbmy02fspb8inxanx-liveviewer

[shana@aya:~/programming/engine]$ cat /tmp/ran
ssh kobuta.production.tsuru.capital -x PATH=/nix/var/nix/profiles/default/bin:"$PATH" nix-store --serve --write
kobuta.production.tsuru.capital
-x
PATH=/nix/var/nix/profiles/default/bin:"$PATH"
nix-store --serve --write

This works. Now I replace nix with 2.15 instead and it no longer works.

[shana@aya:~/programming/engine]$ PATH=/tmp/bin:"$PATH" NIX_SSHOPTS='PATH=/nix/var/nix/profiles/default/bin:"$PATH"' $(nix-build --no-out-link ~/programming/nixpkgs -A nixVersions.nix_2_15)/bin/nix copy --from ssh://kobuta.production.tsuru.capital /nix/store/ch8jsr2qdymmz0xqbmy02fspb8inxanx-liveviewer
bash: -oPermitLocalCommand=yes: command not found
error: failed to start SSH connection to 'kobuta.production.tsuru.capital'

[shana@aya:~/programming/engine]$ cat /tmp/ran
ssh kobuta.production.tsuru.capital -x PATH=/nix/var/nix/profiles/default/bin:"$PATH" -oPermitLocalCommand=yes -oLocalCommand=echo started nix-store --serve --write
kobuta.production.tsuru.capital
-x
PATH=/nix/var/nix/profiles/default/bin:"$PATH"
-oPermitLocalCommand=yes
-oLocalCommand=echo started
nix-store --serve --write

Steps To Reproduce

Just try using PATH with NIX_SSHOPTS on nix 2.15

Expected behavior

I'd expect it to continue to work...

nix-env --version output

2.15.0

Additional context

Quick search in history points to #8018, notably https://github.com/NixOS/nix/pull/8018/commits/5291a82cd9b9d8d7cd6b8338a5224c94c6f23eb7

I'm guessing this doesn't work as it now thinks that the -o stuff is a command, not nix-store command. Maybe NIX_SSHOPTS should be appended, not pre-pended?

Priorities

Add :+1: to issues you find important.

[thufschmitt]

I'm guessing this doesn't work as it now thinks that the -o stuff is a command, not nix-store command. Maybe NIX_SSHOPTS should be appended, not pre-pended?

That sounds reasonable. Care to open a PR for that? (Using it that way is a bit of a hack though, and I wouldn't make any guaranty about it not breaking again in the future).

Note that for your original issue (which is utterly annoying, I'm honestly quite angry at #1078 still being such a thing) you can also add ?remote-program=/run/current-system/sw/bin/nix-store (or nix-daemon for ssh-ng) to the store URL

[Fuuzetsu]

@thufschmitt I tried remote-program and it works with ssh:// though not ssh-ng:// (see at the bottom, :shrug:) thank you for pointing me to that. This is much better than the PATH hack.

I think this should be very much louder stated... somewhere. At least I can't find the reference to it in #1078! Sadly it only works for nix copy and not nix-copy-closure that I can tell but I think most people on nix 2.15 can enable nix-command even if only for just this case. I guess there is some use case where this isn't possible but it's not mine.

That sounds reasonable. Care to open a PR for that?

I can try but it's going be be inefficient. If someone can do it in 5 minutes, that's probably better. If there are no takers, I can spend however long trying to set it up and verify it works.

[shana@aya:~/programming/engine]$ ./bin/viewer # ssh-ng version
warning: Git tree '/home/shana/programming/engine' is dirty
error: no operation specified
Try '/nix/var/nix/profiles/default/bin/nix-store --help' for more information.
error: cannot open connection to remote store 'ssh-ng://kobuta.production.tsuru.capital': error: unexpected end-of-file

[shana@aya:~/programming/engine]$ ./bin/viewer # ssh version
warning: Git tree '/home/shana/programming/engine' is dirty
<snip, works here>

[thufschmitt]

I can try but it's going be be inefficient. If someone can do it in 5 minutes, that's probably better. If there are no takers, I can spend however long trying to set it up and verify it works.

Fair enough, I've opened #8303 for that.

I think this should be very much louder stated... somewhere. At least I can't find the reference to it in https://github.com/NixOS/nix/issues/1078!

Indeed. @balsoft opened https://github.com/NixOS/nix/pull/6628 for that, but the reviews got it lost in over-engineering so it never landed. We should resurrect it (probably in a simpler form).

it only works for nix copy and not nix-copy-closure

Mh, it should. nix-copy-closure just does some nasty string concatenation to build an ssh:// url out of the given host, so we can pass arbitrary arguments to it. At least I just tried nix-copy-closure --to localhost\?remote-program=/bin/sh /run/current-system and it does try to run sh as the remote program.

[georgefst]

Note that for your original issue (which is utterly annoying, I'm honestly quite angry at #1078 still being such a thing) you can also add &remote-program=/run/current-system/sw/bin/nix-store (or nix-daemon for ssh-ng) to the store URL

Thanks for this! This is the nicest workaround I've found for #1078.

By the way, & appears to be a typo, as I've found that only ? works, as you used in your later comment. It might be helpful to correct this for the sake of future readers.

[thufschmitt]

& appears to be a typo, as I've found that only ? works, as you used in your later comment.

It's not entirely a typo. The store urls are urls(ish) as their name indicate, and the parameters are passed as url parameters (?param1=value2&param2=value2&param3=value3). So it depends whether you already have parameters. But yeah, it's confusing indeed, I'll fix that in the comment.

[FlafyDev]

Does anybody know how to set remote-program decoratively with nix.buildMachines(or any other way)?

[malteneuss]

just wanted to mention the needed \? escaping for remote-program:

nix store info --store ssh://<user>@<ip>\?remote-program=/nix/var/nix/profiles/default/bin/nix-store
# older Nix package manager versions:
nix store ping --store ssh://<user>@<ip>\?remote-program=/nix/var/nix/profiles/default/bin/nix-store

[TRVRStash]

Was there ever any movement on this? I just upgraded from nix 2.14 to 2.18.2 and it seems like this is still happening for me. Any tips to unblock? Been working on an ubuntu machine and trying to copy closure onto a NixOS machine.