Connection via ssh fails after update to Unison 2.53

[Ede123]

Hi all,

I'm just in the process of updating from (old) Unison 2.48.3 to Unison 2.53.2.

Unfortunately it seems I can't connect to the remote server via ssh anymore.

I used to rely on interactive password autentication entering the password on the command line. The new version shows a prompt, but I don't seem to be able to authenticate via this prompt. I assume it's asking for a password, but tbh it's not even clear whether it asks for a user name / password / neither:

Any ideas?

Unfortunately there's also no debug output anymore (no console output like in earlier versions), so I'm unsure where to start looking for answers.

[Ede123]

P.S. If I enter the ssh command manually on the command line (cmd.exe) I reach the server on the remote end, so this does not seem to be a fundamental issue:

[tleedjarv]

You are most likely using an incompatible ssh implementation (perhaps a one compiled with Cygwin?). You don't provide all the details but I can see that you are using Windows. Please post output of ssh -V. If you are on Windows 10 or newer, ssh is included with Windows. It is better to try that one.

Next, you can try the current release candidate. Although unlikely, it could fix your issue. Download from https://github.com/bcpierce00/unison/actions/runs/4788914771 (file name ending with -publish)

Unfortunately there's also no debug output anymore (no console output like in earlier versions), so I'm unsure where to start looking for answers.

Debugging output is enabled only when you use the debug preference.

[Ede123]

Thanks for the info!

New results:

• CLI version of unison 2.53.2 still works (asks for password interactively)
• SSH version (C:\Program Files\Unison\bin>ssh -V): OpenSSH_8.7p1, OpenSSL 1.1.1l 24 Aug 2021 Edit: This is an MSYS2 version of SSH. Edit2: So yes, cygwin derivate
• Will experiment more with debug option later Edit: Unfortunately no new info here...

[tleedjarv]

That's the issue, MSYS2 and Cygwin OpenSSH does not work currently in GUI (don't know if we can make it work but I'll investigate). Try with OpenSSH native Windows binary like the one from C:\Windows\System32\OpenSSH or download from https://github.com/PowerShell/Win32-OpenSSH/releases/

Just to be sure, what do you see when you add -sshargs -v on command line? You will probably see ssh complaining about can't open /dev/tty or something similar.

[Ede123]

Thanks, works now with C:\Windows\System32\OpenSSH\ssh.exe!

I've tried adding sshargs = -v in my profile file, but this seems to make the GUI hang with either of the ssh executables (at least it becomes unresponsive and does not allow interaction at all anymore for as long as I allowed it to run before terminating the process).

One minor note: The prompt is a bit confusing (e.g. it notified me that the RSA key of the server has been added where it obviously does not require any actual input from my end...).

[tleedjarv]

This (the -v thing and confusing promp) is fixed in the next release (currently in release candidate form).

[Ede123]

This (the -v thing and confusing promp) is fixed in the next release (currently in release candidate form).

Ah, great! Just tested the latest build and I can confirm both work nicely now, thanks!

For completeness, here's the comparison of verbose output while establishing a working connection with Windows-"native" OpenSSH and failure of establishing a connection with the MSYS2 OpenSSH:

working:

OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
debug1: Connecting to ###.org [###] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\###/.ssh/id_rsa type -1
debug1: identity file C:\\Users\\###/.ssh/id_rsa-cert type -1
debug1: identity file C:\\Users\\###/.ssh/id_dsa type -1
debug1: identity file C:\\Users\\###/.ssh/id_dsa-cert type -1
debug1: identity file C:\\Users\\###/.ssh/id_ecdsa type -1
debug1: identity file C:\\Users\\###/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\###/.ssh/id_ed25519 type -1
debug1: identity file C:\\Users\\###/.ssh/id_ed25519-cert type -1
debug1: identity file C:\\Users\\###/.ssh/id_xmss type -1
debug1: identity file C:\\Users\\###/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version 9.27 FlowSsh: Bitvise SSH Server (WinSSHD) 9.27: free only for personal non-commercial use
debug1: no match: 9.27 FlowSsh: Bitvise SSH Server (WinSSHD) 9.27: free only for personal non-commercial use
debug1: Authenticating to ###.org:22 as '###'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:###
debug1: Host '###.org' is known and matches the RSA host key.
debug1: Found key in C:\\Users\\###/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
debug1: Will attempt key: C:\\Users\\###/.ssh/id_rsa 
debug1: Will attempt key: C:\\Users\\###/.ssh/id_dsa 
debug1: Will attempt key: C:\\Users\\###/.ssh/id_ecdsa 
debug1: Will attempt key: C:\\Users\\###/.ssh/id_ed25519 
debug1: Will attempt key: C:\\Users\\###/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-1.3.132.0.10,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
debug1: kex_input_ext_info: global-requests-ok (unrecognised)
debug1: kex_input_ext_info: no-flow-control (unrecognised)
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: C:\\Users\\###/.ssh/id_rsa
debug1: Trying private key: C:\\Users\\###/.ssh/id_dsa
debug1: Trying private key: C:\\Users\\###/.ssh/id_ecdsa
debug1: Trying private key: C:\\Users\\###/.ssh/id_ed25519
debug1: Trying private key: C:\\Users\\###/.ssh/id_xmss
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such file or directory
###@###.org's password: 

non-working:

OpenSSH_8.7p1, OpenSSL 1.1.1l  24 Aug 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to ###.org [###] port 22.
debug1: Connection established.
debug1: identity file /home/###/.ssh/id_rsa type 0
debug1: identity file /home/###/.ssh/id_rsa-cert type -1
debug1: identity file /home/###/.ssh/id_dsa type -1
debug1: identity file /home/###/.ssh/id_dsa-cert type -1
debug1: identity file /home/###/.ssh/id_ecdsa type -1
debug1: identity file /home/###/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/###/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/###/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/###/.ssh/id_ed25519 type -1
debug1: identity file /home/###/.ssh/id_ed25519-cert type -1
debug1: identity file /home/###/.ssh/id_ed25519_sk type -1
debug1: identity file /home/###/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/###/.ssh/id_xmss type -1
debug1: identity file /home/###/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.7
debug1: Remote protocol version 2.0, remote software version 9.27 FlowSsh: Bitvise SSH Server (WinSSHD) 9.27: free only for personal non-commercial use
debug1: compat_banner: no match: 9.27 FlowSsh: Bitvise SSH Server (WinSSHD) 9.27: free only for personal non-commercial use
debug1: Authenticating to ###.org:22 as '###'
debug1: load_hostkeys: fopen /home/###/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:###
debug1: load_hostkeys: fopen /home/###/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '###.org' is known and matches the RSA host key.
debug1: Found key in /home/###/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/###/.ssh/id_rsa RSA SHA256:###
debug1: Will attempt key: /home/###/.ssh/id_dsa 
debug1: Will attempt key: /home/###/.ssh/id_ecdsa 
debug1: Will attempt key: /home/###/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/###/.ssh/id_ed25519 
debug1: Will attempt key: /home/###/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/###/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ecdsa-sha2-1.3.132.0.10,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa>
debug1: kex_input_ext_info: global-requests-ok (unrecognised)
debug1: kex_input_ext_info: no-flow-control (unrecognised)
debug1: kex_input_ext_info: delay-compression (unrecognised)
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/###/.ssh/id_rsa RSA SHA256:###
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/###/.ssh/id_dsa
debug1: Trying private key: /home/###/.ssh/id_ecdsa
debug1: Trying private key: /home/###/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/###/.ssh/id_ed25519
debug1: Trying private key: /home/###/.ssh/id_ed25519_sk
debug1: Trying private key: /home/###/.ssh/id_xmss
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: 
debug1: Next authentication method: keyboard-interactive

debug1: Authentications that can continue: 
debug1: Next authentication method: password
debug1: No more authentication methods to try.
###@###.org: Permission denied ().

[tleedjarv]

Thank you for testing. I have some good news.

I've investigated the issue with MSYS2/Cygwin ssh a bit. It seems to be a bug in Cygwin but I may be able to work around it.

Background

Unison (a native Windows binary) uses Windows pseudoconsole (conpty) to start ssh (Cygwin-linked binary) and at the same time redirects its stdin, stdout and stderr for its own use. conpty provides a controlling terminal. Cygwin apparently checks for existence of a controlling terminal only by checking stdin, stdout and stderr. These all being redirected, Cygwin decides there is no controlling terminal and does not create /dev/tty (likewise, all POSIX API functions related to controlling terminals will fail). I would guess the correct way for Cygwin to check the existence of a controlling terminal (Windows console) is to try to open "CONOUT$" and/or "CONIN$" and see if they succeed.

openssh reads the interactive auth password from /dev/tty. Long story short, since there is no /dev/tty, there is no interactive auth and the result is seen in comments above.

How come the Windows port of openssh works then? It works precisely because there is no /dev/tty in Windows, so the port directly uses the Windows console (which in this case is provided by conpty).

How did MSYS2/Cygwin ssh work with Unison before conpty was used? It worked because stderr was not redirected and was connected to the Windows console. Cygwin detects existence of a controlling terminal on any of stdin, stdout, stderr so that was sufficient.

Why does the new implementation need to redirect stderr when the previous one didn't? The statement above wasn't quite correct. stderr was redirected, just to the Windows console as it would have been without redirection. Everything still worked just as if stderr hadn't been redirected. The new implementation redirects stderr to the Windows pseudoconsole but to Cygwin this looks like a pipe, not like the native Windows console.

Potential solution/workaround

Since we can't just not redirect stdin and stdout (that's how ssh can be used as a communication tunnel), I investigated further if something could be done with stderr.

I found a rather important remark from Microsoft docs:

Console processes must start with the standard handles filled or they will be filled automatically with appropriate handles to a new console. [...] standard handles are always replaced with console handles unless STARTF_USESTDHANDLES was specified during process creation. If the existing value of the standard handle is NULL, or the existing value of the standard handle looks like a console pseudohandle, the handle is replaced with a console handle. [...] STARTF_USESTDHANDLES to create a console process, standard handles will not be replaced unless the existing value of the standard handle is NULL or a console pseudohandle.

It turns out that we can redirect stdin and stdout while leaving stderr to be populated by Windows. I tried setting stderr to NULL and have Windows attach it to the pseudoconsole for the child process (that would be ssh). This way we still get stderr pointing to where we want it to point and as a bonus it should look like a real console to Cygwin.

This seems to work, at first glance. I haven't done more testing to see if everything works and if it didn't break anything else. There is one immediately obvious glitch. User input is echoed to the console output and thus displayed back to the user in GUI. This is not a major issue though and, importantly, it does not affect password entry because openssh is wise enough to turn off echoing for that.

This fix/workaround will not be in the next release but if it works out well then it will probably be included in a future release.

[tleedjarv]

After some further investigation, I've decided that the glitches are not going away. ConPTY is, as Microsoft themselves put it, a pseudoconsole not a pseudoterminal, so it's primary purpose is to take after Windows console. It means that it's primarily native Windows applications that are supposed to run in it, not native Unix-y application (which Cygwin's openssh undoubtedly is). All that introduces some weirdness and inflexibility. This may and probably will change slowly over time.

That doesn't mean that the workaround won't make it in some future release. @Ede123 I can provide you a test build if you would be interested in testing with MSYS2 ssh. Would you be willing to test?

[Ede123]

I can test, but I'm perfectly happy with using the native ssh client (i.e. I don't see a reason to use a cygwin-like ssh here; it's simply what previously happened to end up on my PATH and used to work). Most likely this works better anyway.

If you provide the test build I'll certainly have a look, though, to see if it would be suitable to land in stable unison eventually.

Otherwise we could just mention the fact that cygwin-provided ssh does not work somewhere in the documentation, so people are aware.

Your choice - I don't want you to put more effort than you want into some edge case.

[tleedjarv]

Yes, your current setup should work better anyway so you should stick to it. But if the fix/workaround works in your setup (which is confirmed to be non-working before) then that's a good confirmation that it's not completely bogus and could land in stable eventually.

If you'd like to help with testing (but no obligation), you can download a test build here https://github.com/tleedjarv/unison/actions/runs/4823348363 (scroll down to see files, pick whichever you want; names ending in -publish have DLLs included but none of those have changed). It is otherwise 2.53.3, with just this patch applied. Try with both the MSYS2 ssh and Windows ssh.

[Ede123]

Tests confirm that this build works with both, the MSYS2 ssh as well as the Windows ssh! :-)

[tleedjarv]

Thank you for testing @Ede123. It seems to me that this fix/workaround might work better than and is less risky than I expected, so it will likely be included in the next major version.