SSH through CDN

[karnal222]

I was following this tutorial: https://kernal.eu/posts/ssh-over-websocket/ Version was the latest linux amd 64 static release. Instead of Cloudflare I tested AWS Cloudfront. If I do a normal site to site connection without CDN then ssh+websocat works. However if I use the CDN in between and on the client do:

ssh -o "ProxyCommand=/sbin/websocat --binary ws://bla.cloudfront.net/ -vvvvv" ubuntu@bla.cloudfront.net -vvv

I get:

[DEBUG websocat::sessionserve] Serving Stdio to WsClient("ws://d164xsza3zusai.cloudfront.net/") with Options { websocket_text_mode: false, websocket_protocol: None, websoc ket_reply_protocol: None, udp_oneshot_mode: false, udp_broadcast: false, udp_multicast_loop: false, udp_ttl: None, udp_join_multicast_addr: [], udp_join_multicast_iface_v4 : [], udp_join_multicast_iface_v6: [], udp_reuseaddr: false, unidirectional: false, unidirectional_reverse: false, max_messages: None, max_messages_rev: None, exit_on_eof: false, oneshot: false, unlink_unix_socket: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_h eaders: [], custom_reply_headers: [], websocket_version: None, websocket_dont_close: false, websocket_ignore_zeromsg: false, one_message: false, no_auto_linemode: false, b uffer_size: 65536, broadcast_queue_len: 16, read_debt_handling: Silent, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: false, n o_exit_on_zeromsg: false, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, socks_destination: None, auto_socks5: None, so cks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None, request_uri: No ne, request_method: None, request_headers: [], autoreconnect_delay_millis: 20, ws_text_prefix: None, ws_binary_prefix: None, ws_binary_base64: false, ws_text_base64: false } [DEBUG websocat::stdio_peer] get_stdio_peer (async) [DEBUG websocat::stdio_peer] Setting stdin to nonblocking mode [DEBUG websocat::stdio_peer] Setting stdout to nonblocking mode [DEBUG websocat::stdio_peer] Installing signal handler [INFO websocat::ws_client_peer] get_ws_client_peer [DEBUG websocat::stdio_peer] restore_blocking_status [DEBUG websocat::stdio_peer] Restoring blocking status for stdin [DEBUG websocat::stdio_peer] Restoring blocking status for stdout websocat: WebSocketError: Received unexpected status code (400 Bad Request) [DEBUG websocat::stdio_peer] restore_blocking_status [DEBUG websocat::stdio_peer] Restoring blocking status for stdin [DEBUG websocat::stdio_peer] Restoring blocking status for stdout websocat: error running kex_exchange_identification: Connection closed by remote host

On the remote end:

sudo websocat --binary ws-l:127.0.0.1:8022 tcp:127.0.0.1:22 -vvv &

Websocat gives an error:

[INFO websocat::net_peer] Incoming TCP connection from Some(V4(127.0.0.1:48418)) [DEBUG websocat::sessionserve] Underlying connection established [INFO websocat::sessionserve] Serving 1 ongoing connections [DEBUG websocat::readdebt] Fullfulling the debt of 135 bytes [DEBUG websocat::trivial_peer] LiteralPeer debt [DEBUG websocat::trivial_peer] LiteralPeer finished [DEBUG websocat::my_copy] zero len [DEBUG websocat::my_copy] read_done [DEBUG websocat::my_copy] done websocat: WebSocketError: I/O failure

From tcpdump I could see:

0x0070: 616d 617a 6f6e 6177 732e 636f 6d0d 0a43 amazonaws.com..C 0x0080: 6f6e 6e65 6374 696f 6e3a 2075 7067 7261 onnection:.upgra 0x0090: 6465 0d0a 5570 6772 6164 653a 2077 6562 de..Upgrade:.web 0x00a0: 736f 636b 6574 0d0a 5573 6572 2d41 6765 socket..User-Age 0x00b0: 6e74 3a20 416d 617a 6f6e 2043 6c6f 7564 nt:.Amazon.Cloud 0x00c0: 4672 6f6e 740d 0a56 6961 3a20 312e 3120 Front..Via:.1.1. 0x00d0: 6133 3732 6465 3732 6634 6461 6366 3163 a372de72f4dacf1c 0x00e0: 6664 3437 6231 6534 3662 6631 3136 3636 fd47b1e46bf11666 0x00f0: 2e63 6c6f 7564 6672 6f6e 742e 6e65 7420 .cloudfront.net. 0x0100: 2843 6c6f 7564 4672 6f6e 7429 0d0a 582d (CloudFront)..X- 0x0110: 466f 7277 6172 6465 642d 466f 723a 2031 Forwarded-For:.1 0x0120: 332e 3233 332e 3139 382e 3130 310d 0a58 3.233.198.102..X 0x0130: 2d41 6d7a 2d43 662d 4964 3a20 7a6c 3579 -Amz-Cf-Id:.zl5y 0x0140: 4e70 494a 3756 4361 6e39 746f 6162 336c NpIJ7VCan9toab3l 0x0150: 3731 6857 4f4b 326b 3755 5072 3474 726d 71hWOK2k7UPr4trm 0x0160: 6a46 6a65 3871 5679 4e58 7559 7664 6663 jFje8qVyNXuYvdfc 0x0170: 3541 3d3d 0d0a 0d0a 5A==.... 15:33:39.232608 IP 172.26.8.156.80 > 70.132.7.132.28236: Flags [.], ack 325, win 974, options [nop,nop,TS val 3737434823 ecr 2929295091], length 0 0x0000: 4500 0034 a20e 4000 4006 95f7 ac1a 089c E..4..@.@....... 0x0010: 4684 0784 0050 6e4c 184f b2a6 3588 0324 F....PnL.O..5..$ 0x0020: 8010 03ce 02e5 0000 0101 080a dec4 bac7 ................ 0x0030: ae99 7ef3 ..~. 15:33:39.233210 IP 172.26.8.156.80 > 70.132.7.132.28236: Flags [P.], seq 1:230, ack 325, win 974, options [nop,nop,TS val 3737434824 ecr 2929295091], length 229: HTTP: HTT P/1.1 400 Bad Request 0x0000: 4500 0119 a20f 4000 4006 9511 ac1a 089c E.....@.@....... 0x0010: 4684 0784 0050 6e4c 184f b2a6 3588 0324 F....PnL.O..5..$ 0x0020: 8018 03ce 03ca 0000 0101 080a dec4 bac8 ................ 0x0030: ae99 7ef3 4854 5450 2f31 2e31 2034 3030 ..~.HTTP/1.1.400 0x0040: 2042 6164 2052 6571 7565 7374 0d0a 5365 .Bad.Request..Se 0x0050: 7276 6572 3a20 6e67 696e 782f 312e 3138 rver:.nginx/1.18 0x0060: 2e30 2028 5562 756e 7475 290d 0a44 6174 .0.(Ubuntu)..Dat 0x0070: 653a 2054 6875 2c20 3234 2044 6563 2032 e:.Thu,.24.Dec.2 0x0080: 3032 3020 3135 3a33 333a 3339 2047 4d54 020.15:33:39.GMT 0x0090: 0d0a 436f 6e74 656e 742d 5479 7065 3a20 ..Content-Type:. 0x00a0: 7465 7874 2f70 6c61 696e 0d0a 5472 616e text/plain..Tran 0x00b0: 7366 6572 2d45 6e63 6f64 696e 673a 2063 sfer-Encoding:.c 0x00c0: 6875 6e6b 6564 0d0a 436f 6e6e 6563 7469 hunked..Connecti 0x00d0: 6f6e 3a20 6b65 6570 2d61 6c69 7665 0d0a on:.keep-alive.. 0x00e0: 0d0a 3263 0d0a 4f6e 6c79 2057 6562 536f ..2c..Only.WebSo 0x00f0: 636b 6574 2063 6f6e 6e65 6374 696f 6e73 cket.connections 0x0100: 2061 7265 2077 656c 636f 6d65 2068 6572 .are.welcome.here

For whatever reason, I get a 400 Bad Request error through the CDN.

Any ideas what could be the issue or how to debug this further?

[vi]

HTTP/1.1 400 Bad Request
Server: nginx/1.18.0...

Looks like the request goes not only though the CloudFront, but also though Nginx.

You need to configure Nginx properly to work with Websockets.

[karnal222]

Yes nginx is already configured in a similar way as in your link or the tutorial I provided. My nginx.conf:

server { listen 80; root /var/www/html; index index.html index.htm index.nginx-debian.html; server_name something.cloudfront.net; location / { proxy_read_timeout 1d; proxy_send_timeout 1d; proxy_pass http://127.0.0.1:8022; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection \"upgrade\"; } }

[vi]

Can you:

1. Try connection the connection to go though Nginx, but not CloudFront? Does it work then?
2. Capture tcpdump -w file (not just the output text above) and attach it?

Is the problem reproducible every time or there are also some successful connection attempts?

Is the problem reproducible if you insert some intermediate TCP forwarder before websocat (e.g. socat tcp-l:7022,fork,reuseaddr tcp:127.0.0.1:8022) and direct Nginx to it?

[karnal222]

1. If I do everything without CloudFront and only nginx + websocat on the remote end it works perfectly. I can ssh into the machine. The problem is 100% reproducible either it works all of the time or, if using CloudFront, it fails 100% of the time with always the same error.

2. Attached is the server side of an unsuccessful connection through CloudFront. I can also post others if required.

I'll try to fiddle a bit with socat.

[vi]

1. Is it possible to attempt to direct CloudFront to Websocat directly?
2. What version of Websocat do you use on server?
3. Can you try to test older versions (for example, versions 1.2.0 and 0.4.0)? Correct command line for version 0.4.0 is like this: websocat_nossl_0.4_i686-unknown-linux-musl l-ws:127.0.0.1:8022 tcp:127.0.0.1:2222. You can find pre-built old versions on Github releases.

[karnal222]

1. When I use

   sudo websocat --binary ws-l:0.0.0.0:80 tcp:127.0.0.1:22 -vvv &

The error is the same (400 Bad request, Only websocket connections are welcome here). So the problem is probably not nginx. Maybe CloudFront mangles some Header or other stuff which websocat expects in some other form? I see some additional and missing headers in case I use no CloudFront and the connection is working (pcap attached)

2. Client and Server versions are the same: https://github.com/vi/websocat/releases/download/v1.6.0/websocat_amd64-linux-static
3. Will try.

[vi]

Maybe CloudFront mangles some Header

Now I see: Sec-WebSocket-Version and Sec-WebSocket-Key headers required for WebSocket connection establishment are missing.

Maybe you need to manually enable certain headers. CloudFront's documentation mentions it should be compatible with WebSockets without additional configuration, so maybe you explicitly configured it to drop some headers?

[karnal222]

Yeah in theory websockets should work with cloudfront. I tried to set the header manually (which I guess is not the correct thing to do) and the error looks a bit different on the client:

websocat: WebSocket response error: Sec-WebSocket-Accept is invalid

pcap:

On the server there does not seem to be an error:

[INFO websocat::net_peer] Incoming TCP connection from Some(V4(127.0.0.1:36364)) [DEBUG websocat::sessionserve] Underlying connection established [INFO websocat::sessionserve] Serving 1 ongoing connections [INFO websocat::ws_server_peer] Incoming connection to websocket: / [DEBUG websocat::ws_server_peer] Incoming { version: Http11, subject: (Get, AbsolutePath("/")), headers: Headers { Upgrade: websocket , Connection: Upgrade , Host: 127.0.0.1:8022 , User-Agent: Amazon CloudFront , Via: 1.1 5066297e571270218f6c9f8fb4fdd813.cloudfront.net (CloudFront) , X-Forwarded-For: 65.0.93.44 , X-Amz-Cf-Id: OvKSiW9AFIcSz5nIsUdMnZc5h55xU5Jkm6uetSGg7z4bKPSQW70U4g== , Sec-WebSocket-Key: dhpc9Q732h/9d72CNGvZ7A== , } } [DEBUG websocat::ws_server_peer] Headers { } [DEBUG websocat::ws_server_peer] Headers { Sec-WebSocket-Accept: R68+WKf5Jt2eNoBREskuR5bYoZY= , Connection: Upgrade , Upgrade: websocket , } [INFO websocat::ws_server_peer] Upgraded [INFO websocat::net_peer] Connected to TCP 127.0.0.1:22 [DEBUG websocat::net_peer] We have a winner. Disconnecting losers. [DEBUG websocat::ws_peer] incoming None [DEBUG websocat::my_copy] BrokenPipe: read_done [DEBUG websocat::my_copy] done [INFO websocat::sessionserve] Forward finished [DEBUG websocat::sessionserve] Forward shutdown finished [DEBUG websocat::my_copy] zero len [DEBUG websocat::my_copy] read_done [DEBUG websocat::my_copy] done [INFO websocat::sessionserve] Reverse finished [DEBUG websocat::sessionserve] Reverse shutdown finished [DEBUG websocat::ws_peer] drop WsWriteWrapper [INFO websocat::sessionserve] Both directions finished

I'll take a closer look at the headers (tomorrow^^, thx for help up to now :) )

[vi]

I tried to set the header manually (which I guess is not the correct thing to do)

Do you mean setting it to a fixed value instead of client-supplied? If yes then this won't work. It is a protection against WebSocket clients (i.e. web pages without any special permissions from users) connecting to something other than WebSocket servers.

Sec-WebSocket-Accept is invalid

Sec-WebSocket-Accept is (sort-of-cryptographically) derived from Sec-WebSocket-Key.

---

You need to investigate and debug why Sec-WebSocket-Key is getting lost on the way through the CloudFront.

[karnal222]

Looks like it is working with: https://nickzamosenchuk.medium.com/configure-amazon-cloudfront-cdn-for-websocket-connection-43c44b3f877c :)

[vi]

So is the issue resolved?

[karnal222]

Yes :)