Messages / packets get mangled when handling multiple connections / forwards

[wolfman42]

Describe the bug Looks like there is a fundamental concurrency issue when handling multiple connections / forwards where messages get mangled along the way. I was able to reproduce the problem in a simple setup with an echo server and multiple echo clients. However, i discovered the issue in a more complex setup where I run multiple SSH connections via SOCKS5 proxy to multiple hosts. The SSH connections weren't stable and kept breaking down with errors like "bad packet length".

The same issue happens when using port-forwarding configured via the --local-to-remote option. So, I don't think the issue has anything to do with the SOCKS5 implementation itself. Rather, the established connection between a client and server seems to have concurrency issues with other / parallel established connections.

To Reproduce test-python-scripts.zip

• Launch wstunnel server: ./wstunnel server ws://127.0.0.1:8888
• Launch wstunnel client: ./wstunnel client -L tcp://9191:127.0.0.1:8181 ws://localhost:8888
• Launch echo-server.py
• Launch echo-client.py

After a few seconds, the first client will exit / stop, complaining that the response didn't match the received response.

I also included launch-client-wstunnel.py which launches 10 wstunnel clients, each listening on a different port. The result is the same when using it. (Note: You'll have to modify the echo-client.py as documented there)

Expected behavior Concurrent connections to the same as well as different hosts by multiple clients need to not mangle IP packets / messages.

Screenshots The echo client will print output like this after a few seconds of running:

Received bad echo response Client port 9191 EDiJlislCcOuyDDTENpoAT8xr1PKq9TaaFgeBzQq14OHZOaS0cbybqj9Ka1AeDszeiGK8LF3L6RexLnLMoeehanUlvjIE50P5cXGgT1YQnqhtuPnvKMmKI9bF3s1Eiy99yYfgaxbny1eJCLHXmusRVX3I8mgC9KCcBxI3yMkmXQOrlnNCtO48qzmM4KIN4iqpQFrNfOyOc48unfWTRQ7UyYW4NmUpudjR7w7CW8JJ7KotWzKUTZOdnSwVFZAp1e7 for message Client port 9191 PZs9Wb4FscQkn6gL2sGbT9UXKulc2NhOJls4SCCKn5jybzqBpJBYUeKhSgyMJoR1FE9sEK0uvEv7SmR82UBBFQyNPama5qJTt4ZOWrcYmAbpB2M1KrmYVu1xUCcEnmorlpSxd49YnQrZJYB4UQQdzNZkhxEeLojBTVKGqa9PzpboBsPZUhqhGNLfuEeFAke7PrtgTIvxkBtCZd9MG7nxWwyzlBFYc7uurtqJoBmnJvTxN2Az62bSv0af8lU0GrVC

Desktop (please complete the following information):

• OS: Windows
• Version 9.1.0 of wstunnel

Additional context none

[erebe]

Hi,

Thank you for the reproduce case, it is really helpful :pray:

It turns out you are right, I managed to reproduce the issue and after investigation it seems linked to the websocket library I am using, and more specifically this issue https://github.com/denoland/fastwebsockets/issues/42

I tried to use another transport protocol (http2) than websocket from a not yet released version, and I can't reproduce the issue in this case. So I am pretty confident that it is the bug linked that is the culprit.

I will try to fork the lib to set a buffer per websocket, and keep you posted. (nothing before end of week i think)

In the meantime and as a workaround, you can start your wstunnel client and server with TOKIO_WORKER_THREADS=1 env variable set. it will use a single thread, and in this case i didn't manage to trigger the bug

Thanks for the report and the repro case :love_letter:

[cattyhouse]

looking forward to the http2 version, i guess it's less overhead than websocket?

[erebe]

You will be disappointed I think @cattyhouse :)

I wanted this feature, in case where websocket connection were forbidden. They are fairly trivial to detect and block in some scenario. So wanted to add http2 as transport protocol (which support streaming) to make it harder to block.

While it works, it is sadly harder to make it reliable, mostly due to reverse-proxy as they tend to buffer requests before passing them to the app. Also, most reverse proxy don't speak http2 to the upstream, they turn http2 into http1 connection.

Also websocket has less overhead/complexity than http2, so you will not see any perf improvement

[erebe]

FYI @wolfman42 , I have a dirty patch that works and fix the issue. I need to clean it up a bit, but as I will not have much free time this end of week, a new release will most likely land only next week;

[wolfman42]

Thanks a lot @erebe . The workaround setting TOKIO_WORKER_THREADS=1 already works for me in the meantime.

Apologies for piling on. However, since you're already working that area of the code, what do you think about this new enhancement request? #217

Having long-lived connections is an important use case for my scenario. Refreshing the authorization header would help greatly to make that happen.

[erebe]

It has been easier than I thought. Would you mind trying this release https://github.com/erebe/wstunnel/releases/tag/v9.2.0 and let me know if it is ok for you ?

[cattyhouse]

can you document https scheme usage?

[erebe]

I plan to do it for next week @cattyhouse. But if you want to toy with it. You can start your server as usual wstunnel server wss://xxxx and just change how you start your client. wstunnel client .. https://xxxx instead of wss://xxxx`

The server is going to pick automatically http2, only the client need to specify if it wants to use websocket or http2 as transport protocol. (You may need to add extra header content-type or content-length if you want to bypass some firewall, beware as I mentionned of proxy that buffer requests)

[cattyhouse]

I plan to do it for next week @cattyhouse. But if you want to toy with it. You can start your server as usual wstunnel server wss://xxxx and just change how you start your client. wstunnel client .. https://xxxx instead of wss://xxxx`

The server is going to pick automatically http2, only the client need to specify if it wants to use websocket or http2 as transport protocol. (You may need to add extra header content-type or content-length if you want to bypass some firewall, beware as I mentionned of proxy that buffer requests)

Thanks, was using https:// (pure guess) on server side, not working, that's why i asked about usage.

i just run speedtest -s 7190 proxied via wstunnel client tproxy, wss:// can generate much higher download throughput than https://. plus while running speedtest against https:// scheme, wstunnel server got killed in the middle, the vps server has about 1GB RAM.

journalctl -eb -p4

tokio-runtime-w invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0

....

Out of memory: Killed process 216132 (wstunnel) total-vm:850704kB, anon-rss:363564kB, file-rss:1592kB, shmem-rss:0kB, UID:0 pgtables:1700kB oom_score_adj:0

[erebe]

Told you about disappointment :0

Will check next week for the oom.

[erebe]

@cattyhouse if you re-download the version 9.2.0 you should be good for the oom