mux_enable_ssh_agent=true disables private key authentication w/ OpenSSH_for_Windows

andreimaxim commented 1 month ago

What Operating System(s) are you seeing this problem on?

Windows

Which Wayland compositor or X11 Window manager(s) are you using?

N/A

WezTerm version

Nightly

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

Yes, and I updated the version box above to show the version of the nightly that I tried

Describe the bug

When running the Nightly version on Windows 11 and using ssh.exe to connect to a remote host using a private key, the connection fails with a Permission denied (publickey).

The verbose output:

C:\Users\andre>ssh.exe -T git@github.com -vv
OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug2: resolving "github.com" port 22
debug1: Connecting to github.com [140.82.121.4] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\andre/.ssh/id_rsa type -1
debug1: identity file C:\\Users\\andre/.ssh/id_rsa-cert type -1
debug1: identity file C:\\Users\\andre/.ssh/id_dsa type -1
debug1: identity file C:\\Users\\andre/.ssh/id_dsa-cert type -1
debug1: identity file C:\\Users\\andre/.ssh/id_ecdsa type -1
debug1: identity file C:\\Users\\andre/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\andre/.ssh/id_ecdsa_sk type -1
debug1: identity file C:\\Users\\andre/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file C:\\Users\\andre/.ssh/id_ed25519 type -1
debug1: identity file C:\\Users\\andre/.ssh/id_ed25519-cert type -1
debug1: identity file C:\\Users\\andre/.ssh/id_ed25519_sk type -1
debug1: identity file C:\\Users\\andre/.ssh/id_ed25519_sk-cert type -1
debug1: identity file C:\\Users\\andre/.ssh/id_xmss type -1
debug1: identity file C:\\Users\\andre/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.6
debug1: Remote protocol version 2.0, remote software version babeld-64443082
debug1: compat_banner: no match: babeld-64443082
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to github.com:22 as 'git'
debug1: load_hostkeys: fopen C:\\Users\\andre/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts: No such file
or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,kex-strict-s-v00@openssh.com
debug2: host key algorithms: ssh-ed25519,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
debug1: load_hostkeys: fopen C:\\Users\\andre/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts: No such file
or directory
debug1: load_hostkeys: fopen __PROGRAMDATA__\\ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'github.com' is known and matches the ED25519 host key.
debug1: Found key in C:\\Users\\andre/.ssh/known_hosts:1
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: C:\\Users\\andre/.ssh/id_rsa
debug1: Will attempt key: C:\\Users\\andre/.ssh/id_dsa
debug1: Will attempt key: C:\\Users\\andre/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\andre/.ssh/id_ecdsa_sk
debug1: Will attempt key: C:\\Users\\andre/.ssh/id_ed25519
debug1: Will attempt key: C:\\Users\\andre/.ssh/id_ed25519_sk
debug1: Will attempt key: C:\\Users\\andre/.ssh/id_xmss
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: C:\\Users\\andre/.ssh/id_rsa
debug1: Trying private key: C:\\Users\\andre/.ssh/id_dsa
debug1: Trying private key: C:\\Users\\andre/.ssh/id_ecdsa
debug1: Trying private key: C:\\Users\\andre/.ssh/id_ecdsa_sk
debug1: Trying private key: C:\\Users\\andre/.ssh/id_ed25519
debug1: Trying private key: C:\\Users\\andre/.ssh/id_ed25519_sk
debug1: Trying private key: C:\\Users\\andre/.ssh/id_xmss
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
git@github.com: Permission denied (publickey).

I suspect that the following line shows the actual issue, as it does not appear in the logs of the current release, which does not exhibit this issue:

debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory

After some debugging, I believe that the issue was introduced in 4af418fddd0ed2d9a8861007112006fc657ecbac since setting mux_enable_ssh_agent to false fixes the issue.

To Reproduce

Connect to Github using the Windows SSH client using the Nightly WezTerm build:

ssh.exe -T git@github.com

Configuration

no config

Expected Behavior

The following output is expected:

$ ssh.exe -T git@github.com
Hi andreimaxim! You've successfully authenticated, but GitHub does not provide shell access.

Logs

No response

Anything else?

No response

wez commented 1 month ago

How does WSL2 factor into this? It looks like you are running cmd/pwsh and the native ssh binary?

With mux_enable_ssh_agent = false, what is your SSH_AUTH_SOCK environment set to in your cmd/pwsh session where you run ssh?

I suspect the problem here is really that that version of ssh doesn't support unix domain sockets:

https://github.com/PowerShell/Win32-OpenSSH/issues/1761

I think the approach I'd like to take for this is to maybe try to intelligently select a default for mux_enable_ssh_agent using a simple/cheap/effective probe of some kind that can detect when an existing agent is not reachable via unix domain.

I do want to note that simply defaulting it to false on Windows isn't something I really want to do, as wezterm ssh and other ssh implementations do support unix domain sockets.

andreimaxim commented 1 month ago

@wez sorry, I initially wrote "WSL2" because that's where I noticed it happening, but then I was able to reproduce it in Windows directly (I figured it's easier to test it like that). I've updated the title and removed the reference to WSL.

With mux_enable_ssh_agent = false, what is your SSH_AUTH_SOCK environment set to in your cmd/pwsh session where you run ssh?

It's not set.

I think it's expected for Windows to not set the SSH_AUTH_SOCK environment variable at all (see PowerShell/Win32-OpenSSH#1136), as echo $env:SSH_AUTH_SOCK in Windows Terminal + PowerShell does not return anything as well.

I think the approach I'd like to take for this is to maybe try to intelligently select a default for mux_enable_ssh_agent using a simple/cheap/effective probe of some kind that can detect when an existing agent is not reachable via unix domain.

Would it make sense to enable the SSH agent only if the "SSH_AUTH_SOCK" environment variable is set?

wez commented 1 month ago

Would it make sense to enable the SSH agent only if the "SSH_AUTH_SOCK" environment variable is set?

It's unfortunately not that easy; it is often unset on most systems, when the desktop environment first loads.

I think it's expected for Windows to not set the SSH_AUTH_SOCK environment variable at all (see PowerShell/Win32-OpenSSH#1136), as echo $env:SSH_AUTH_SOCK in Windows Terminal + PowerShell does not return anything as well.

When you first launch windows, is the agent started at login/boot, or does it spawn when ssh.exe is launched?

If the agent is started before wezterm, then it may be possible to try to probe the \.\pipe\openssh-ssh-agent path to decide if that agent is in use. Otherwise, I don't think there is a great option for this.

andreimaxim commented 1 month ago

Windows comes with a SSH Authentication Agent service that will create \.\pipe\openssh-ssh-agent at launch, but it is disabled by default. However, if you use 1Password and enable their SSH agent (and most likely any other software that might act as an SSH Agent), it will also create \.\pipe\openssh-ssh-agent when 1Password starts.

That being said, I'm starting to think that the cleanest solution in this case is to add a warning somewhere (changelog? some FAQ?) about the potential behavior change on Windows due to this new feature.

Thank you very much for your work on WezTerm!

abelcheung commented 1 month ago

I just bumped into the same error with nightly wezterm build on Windows 10 (this is Server but I think the situation is identical to other consumer editions). With wezterm setting up $SSH_AUTH_SOCK for me unnecessarily, and additionally the env variable is set to a non-existing file, causing auth to fail. Unsetting the env variable temporarily fixes auth for me. Besides, the problem is not limited to powershell (core); good ol' cmd.exe also faces the same error.

Probing for existing named pipe without touching $SSH_AUTH_SOCK is a solution for Microsoft port of OpenSSH. This native port enjoys increasing usage and is the default for Windows 10+. However the diversity of other ssh clients can mean more headache.

The bundled ssh from Git for Windows is one such example, as well as all(?) MingW / MSYS derivated ssh builds. Those do use $SSH_AUTH_SOCK -- but they talk via Cygwin-compatible socket file. The env variable is generally managed by Windows users themselves.
Let's just forget putty which uses its own pageant for now

abelcheung commented 1 month ago

BTW I should mention my setup too to show the diversity of possibilities in Windows. Maybe a bit similar to @wez usage of 1Password's agent, but with different software.

Agent: a KeePass plugin (KeeAgent), which opens necessary named pipe and Cygwin socket by itself Keys and passwords: stored within KeePass SSH binary:

In pwsh and cmd: Microsoft port, but with its own agent disabled. $SSH_AUTH_SOCK not set.
In MSYS2 bash: Bundled ssh, $SSH_AUTH_SOCK set to socket file exported via KeeAgent
In WSL bash: Bundled ssh, with $SSH_AUTH_SOCK set by helper script that communicates with named pipe in Windows host

wez / wezterm