Win32 OpenSSH Reverse Connection Tunnelling is not working

[mwestphal]

Server OperatingSystem any linux based os

Client OperatingSystem Windows 10 with Spring Update

What is failing There is definitely a bug in Microsoft OpenSSH implementation from 2018 Spring Update.

How to test it : On the local (Windows 10), install Python3, Putty and make sure ssh is available. Then

1. Run a python http server in a terminal (on port 8000 by default)

   python -m http.server

   2. Create a reverse connection ssh tunnel

   ssh -R 8080:localhost:8000 user@remote

On the remote, connect trough the tunnel with telnet On The remote, connect to it trough telnet

>telnet localhost 8080
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.

The tunnel is still runnning but telnet disconnect almost instantly after the connection.

If you replase ssh by the plink.exe from putty, it works flawlessly and you can connect with a browser.

Expected output

    >telnet localhost 8080
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.

Actual output

    >telnet localhost 8080
    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    Connection closed by foreign host.

[aaronelliotross]

@mwestphal I'm having the same problem, have you learned anything since October about what the problem is ?

[mwestphal]

I'm not working on this anymore, since we consider the Putty work around to be acceptable. Only Win32-OpenSSH devs will be able to help I think.

[manojampalam]

I've tried the following on 1809 update and it seems that reverse tunneling is working fine

Server: Ubuntu Client: Windows

On client

• Run an instance of sshd in debug mode on port psexec -s sshd -d -p 47003
• launch ssh session with reverse port forwarding ssh -R 47010:localhost:47003 targetuser@target

On Server

• ssh into the daemon instance running on client ssh -p 47010 clientuser@localhost

[aaronelliotross]

Thanks for the testing example !

I've updated to 1809 and your example works for me now too.

However, I'm still having trouble getting other connections to work. Here's an example that works in WSL with an ssh built with "OpenSSH_7.6p1 Ubuntu-4ubuntu0.1, OpenSSL 1.0.2n 7 Dec 2017, but not with the ssh in PowerShell "OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5".

On client:

> python -m http.server 43333
Serving HTTP on 0.0.0.0 port 43333 (http://0.0.0.0:43333/) ...

Launch connection with reverse forwarding in PowerShell

> ssh -R 59245:localhost:43333 server

On the server send an HTTP request:

$ echo "GET / HTTP/1.0" | nc localhost 59245
$

The expected output is a HTML response (a directory listing). The actual output is no output at all. An strace show that the connection is made and the string written to the socket. Wireshark shows packets coming through, but the HTTP server on the Client doesn't see the request at all.

On the WSL side, the same commands result in a response:

On Client:

$ ssh -R 59244:localhost:43333 remote

On the Server:

$ echo "GET / HTTP/1.0" | nc localhost 59244
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.7.1
Date: Fri, 16 Nov 2018 15:02:40 GMT
Content-type: text/html; charset=utf-8
Content-Length: 2755
   … snip the rest of the request ...

I'm very new to PowerShell, so while I've been trying to think of anything stupid I'm doing here, I might be missing something very obvious.

Thanks for your help !

[mwestphal]

I tried my example again, and it still fails. No idea why your sshd example works for you.

[june07]

Just a confirmation that I am also seeing this issue... any updates?!

[june07]

More information regarding the issue...

Works fine when limiting to IPv4 using the "-4" flag. Also works fine when using 127.0.0.1 instead of localhost. Fails when using localhost. Should be a big clue to the maintainers of this code base. Seems like a bug to me.

[mwestphal]

Interesting work around, thanks for sharing.

[DuBistKomisch]

Thanks, 127.0.0.1 fixed it for me too. Definitely still a bug in Windows 10 2004.

[HenrikBengtsson]

Can confirm @june07's workaround 🙏

I've been following this issue for several years as part of providing parallel processing in R with remote parallel workers, and this bug has been a blocker ever since. We have had to recommend PuTTY's plink or MinGW's ssh solutions instead of the Windows 10 built-in ssh (https://github.com/HenrikBengtsson/parallelly/issues/7). When using:

ssh -R 12345:localhost:12345 remote.server.org Rscript -e "launchRemoveWorkers()" ...

it would block/fail with a corrupt socket connection. However, thanks to @june07's workarounds (https://github.com/PowerShell/Win32-OpenSSH/issues/1265#issuecomment-637085238) there is now a way forward with the built-in SSH client on Windows 10. I can now basically include the 'localhost' -> '127.0.0.1' workaround internally until properly fixed in the MS Windows client.

Details:

Works fine when limiting to IPv4 using the "-4" flag.

I can confirm that this passing -4 workaround works for our purposes;

ssh -R 12345:localhost:12345 -4 remote.server.org Rscript -e "launchRemoveWorkers()" ...

Also works fine when using 127.0.0.1 instead of localhost.

Confirming; the following works just fine for our needs, which is also the workaround I went with:

ssh -R 12345:127.0.0.1:12345 remote.server.org Rscript -e "launchRemoveWorkers()" ...

Fails when using localhost.

Yup

Should be a big clue to the maintainers of this code base. Seems like a bug to me.

Agree. That's a very good clue for troubleshooting this. I'd bet if this reaches the correct person/developer, it's a quick fix.

[JoakimHagen]

I believe this is not a bug in the implementation of OpenSSH, but rather an issue of Windows' loopback resolution using IPv6 by default and the server not listening for it, hence why -4 and 127.0.0.1 works. See my question and answer on superuser

[SkytAsul]

Still an issue 6 years later, that's crazy.