Using websocat as a websocket proxy

[grepsuzette]

Suppose we would want to create a websocket proxy server using websocat. For instance, client A needs to access ws://somedomain/somewhere. But the websocat to use would be on another computer, acting as a proxy.

What would be the easiest way to do that using websocat? I think it would necessitate another program that would accept the connection, read the target uri and then instantiate websocat with that target uri and a mirror:. Is there a simpler way?

[vi]

So you mean something like

websocat --text ws-l:0.0.0.0:80  ws://somedomain/somewhere

?

It should work as a proxy (accept WebSocket connections on local port 80 on any URL and forward them to somedomain). Note that such proxy would be loosy:

• Binary WebSocket messages get concerted to text WebSocket messages (or vice versa if --binary instead of --text).
• Incoming URL is lost and is converted to fixed URL from command line
• WebSocket pings are terminated and replied by Websocat, not proxied.
• WebSocket version and protocol in the connect request is not proxied.

What do you want to attain with such proxying? Is it for development or for production?

Note that nginx can also proxy WebSocket connections and may be a better tool for this.

[grepsuzette]

websocat --text ws-l:0.0.0.0:80 ws://somedomain/somewhere

Like this but where the url ws://somedomain/somewhere is provided by the connecting client(s). This is for developement purpose.

The way I imagine it it would need a server accepting connections (bash or perl should do, hopefully, no point if it's not simple) and then the incoming socket would need to be plugged to the stdin of a websocat process connected to that url? Or maybe the websocat could be instantiated like your example, and instead of redirecting stdin it would need connecting the incoming client socket to the listening socket on the left-hand side of the websocat.

Had researched nginx capabilities already, but I didn't get it, it seemed different from what I needed. Although I have already some homemade websocket proxy, using highlevel langage (hard to maintain, time-consuming honestly), I was wondering what would be the best way to move to a bash or perl + websocket solution for this hopefully simple tool (at home I'm behind national firewalls and would hate using a vpn or whatever, so this tool would be very general purpose for me).

I'm already using websocat for other kind of codes (for servers) and I really like the unix philosophy behind, along with the powerful options, it makes things simple and fast.

[vi]

Like

websocat -Ee --text ws-l:127.0.0.1:8080 sh-c:'websocat ws://echo.websocket.org$WEBSOCAT_URI'

?

[grepsuzette]

Nice! I knew this thing was powerful :) I'm getting encouraging results so far with:

websocat -Ee --text ws-l:0.0.0.0:8080 sh-c:'websocat ${WEBSOCAT_URI:1}'

In fact it works. As long as ws is used.

Connecting from a remote computer with websocat ws://myserver:8080/ws://echo.websocket.org by removing the leading / we can use that as a URL, and the echo on stdin is received from the other side, just like you suggested!

Now, a problem seems with urls using wss... I would think proxy to target should use its own keys, and client to proxy can regardless use ws:// or wss://, but I'm getting some error.

Take for example websocat -E wss://myserver:8080/wss://stream.binance.com:9443/ws/btcusdt@ticker. Running with -v on the proxy we get those messages:

[INFO  websocat::ws_server_peer] Incoming connection to websocket: /wss://stream.binance.com:9443/ws/btcusdt@ticker
[INFO  websocat::ws_server_peer] Upgraded
websocat: WebSocketError: I/O failure

The culprit is not the initial / as it is removed by ${WEBSOCKET_URI:1}. Client side, I also receive

websocat: WebSocketError: I/O failure
websocat: error running

Does this maybe has to do with the sh-c's websocat needing some configuration for tls? (I'll read some doc and try with --pkcs12-der or something. Edit: but no need for that if socket is not listening, what could it be then?).

[vi]

What is printed if -v is specified to inner websocat?

sh-c:'websocat -v ${WEBSOCAT_URI:1}'

Note that sh-c: starts sh, not bash. Such substring substitution seems to fail for dash, which is often implementation of /bin/sh.

I tried

websocat -Ee --text ws-l:0.0.0.0:8080 exec:/bin/bash --exec-args -c 'websocat -v ${WEBSOCAT_URI:1}'

And both websocat ws://127.0.0.1:8080/ws://echo.websocket.org/ and websocat ws://127.0.0.1:8080/wss://echo.websocket.org/ work reasonably.

May be you also want -k (--insecure)?

[grepsuzette]

I prematurely wrote it didn't work with wss yesterday. I in fact thought so because directly jumped to testing with a specific stream (binance). Not only wss://echo.websocket.org works but also wss://api.bitfinex.com/ws (sending a line { "event":"subscribe", "channel":"ticker", "pair":"BTCUSD" } to subscribe one channel).

So in general the above solution works. Which responds to the question greatly. Thanks about that and for websocat :)

---

A stream that still refuses to be proxied is wss://stream.binance.com:9443/ws/btcusdt@ticker, despite having websocat wss://stream.binance.com:9443/ws/btcusdt@ticker working flawlessly when ran in a shell on the proxy server.

Any flag combination seems useless, even a -v to the internal websocat will not bring more verbosity.

It couldn't be because of the @ character could it?

[vi]

Seem to work for me. On one tab I have:

$ websocat -Ee --text ws-l:0.0.0.0:8080 exec:/bin/bash --exec-args -c 'websocat -v ${WEBSOCAT_URI:1}'
[INFO  websocat::lints] Auto-inserting the line mode
[INFO  websocat::sessionserve] Serving Line2Message(Stdio) to Message2Line(WsClient("wss://stream.binance.com:9443/ws/btcusdt@ticker")) with Options { websocket_text_mode: true, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, unidirectional: false, unidirectional_reverse: false, exit_on_eof: false, oneshot: false, unlink_unix_socket: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [], custom_reply_headers: [], websocket_version: None, websocket_dont_close: false, one_message: false, no_auto_linemode: false, buffer_size: 65536, broadcast_queue_len: 16, read_debt_handling: Warn, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: false, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, socks_destination: None, auto_socks5: None, socks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None }
[INFO  websocat::stdio_peer] get_stdio_peer (async)
[INFO  websocat::stdio_peer] Setting stdin to nonblocking mode
[INFO  websocat::stdio_peer] Setting stdout to nonblocking mode
[INFO  websocat::stdio_peer] Installing signal handler
[INFO  websocat::ws_client_peer] get_ws_client_peer
[INFO  websocat::ws_client_peer] Connected to ws
^C

On the other:

$ websocat ws://127.0.0.1:8080/wss://stream.binance.com:9443/ws/btcusdt@ticker
{"e":"24hrTicker","E":1565866495587,"s":"BTCUSDT","p":"-469.41000000","P":"-4.449","w":"10264.12771849","x":"10550.05000000","c":"10080.62000000","Q":"0.00110200","b":"10072.06000000","B":"0.09200000","a":"10080.62000000","A":"0.05440100","o":"10550.03000000","h":"10697.00000000","l":"9928.10000000","v":"42113.96786000","q":"432263144.84724437","O":1565780095581,"C":1565866495581,"F":168658244,"L":169074884,"n":416641}
{"e":"24hrTicker","E":1565866496615,"s":"BTCUSDT","p":"-469.41000000","P":"-4.449","w":"10264.12679879","x":"10550.05000000","c":"10080.62000000","Q":"0.05440100","b":"10075.45000000","B":"0.01983800","a":"10081.41000000","A":"0.07998300","o":"10550.03000000","h":"10697.00000000","l":"9928.10000000","v":"42114.17672100","q":"432265249.89093299","O":1565780096609,"C":1565866496609,"F":168658244,"L":169074886,"n":416643}
{"e":"24hrTicker","E":1565866497658,"s":"BTCUSDT","p":"-468.63000000","P":"-4.442","w":"10264.12616603","x":"10550.03000000","c":"10081.40000000","Q":"0.10261200","b":"10075.47000000","B":"0.06519100","a":"10081.40000000","A":"0.41089900","o":"10550.03000000","h":"10697.00000000","l":"9928.10000000","v":"42114.28496700","q":"432266334.29338390","O":1565780097654,"C":1565866497654,"F":168658245,"L":169074890,"n":416646}
{"e":"24hrTicker","E":1565866498675,"s":"BTCUSDT","p":"-473.02000000","P":"-4.484","w":"10264.12490778","x":"10550.05000000","c":"10077.01000000","Q":"0.04786000","b":"10075.53000000","B":"0.01983800","a":"10081.39000000","A":"0.00178300","o":"10550.03000000","h":"10697.00000000","l":"9928.10000000","v":"42114.29227200","q":"432266356.28246870","O":1565780098671,"C":1565866498671,"F":168658247,"L":169074896,"n":416650}

[grepsuzette]

Tab 1:

websocat -Ee --text ws-l:0.0.0.0:8080 exec:/bin/bash --exec-args -c 'websocat -v ${WEBSOCAT_URI:1}'
websocat: WebSocketError: I/O failure

Tab 2:

websocat ws://<serverIp>:8080/wss://stream.binance.com:9443/ws/btcusdt@ticker
websocat: WebSocketError: I/O failure
websocat: error running

Where is replaced with the actual IP. A difference is that you seem to have the proxy and the client on the same computer, though I think it should also work for me. It's strange, I'll see if I need maybe to upgrade to the latest version, and maybe turning off SELinux.

[vi]

Another simple diagnostic test.

Proxy tab: websocat -Ee --text ws-l:0.0.0.0:8080 exec:/bin/bash --exec-args -c 'echo websocat -v ${WEBSOCAT_URI:1}' (note the echo)

Client tab:

$ websocat ws://127.0.0.1:8080/wss://stream.binance.com:9443/ws/btcusdt@ticker
websocat -v wss://stream.binance.com:9443/ws/btcusdt@ticker

This is not a proxy, but allows you to easily inspect the actual command line.

[grepsuzette]

Still no luck.

• Proxy host had already version 1.5.0. And SELinux was already disabled.
• Client host had version 1.4.0, upgraded to 1.5.0 (checked version was upgraded with websocat -V).
• I checked previous connections to 9443 were closed with netstat -an | grep 9443, same on the client with grep 8080.

It changes nothing.

Running the previous with "echo" gives this line on stdout: websocat -v wss://stream.binance.com:9443/ws/btcusdt@ticker. Same as you, this seems normal.

Another try (I'll call this "type 1 proxy output"):

websocat -v -Ee --text ws-l:0.0.0.0:8080 exec:/bin/bash --exec-args -c 'websocat -v ${WEBSOCAT_URI:1}'
[INFO  websocat::lints] Auto-inserting the line mode
[INFO  websocat::sessionserve] Serving Message2Line(WsServer(TcpListen(V4(0.0.0.0:8080)))) to Line2Message(Exec("/bin/bash")) with Options { websocket_text_mode: true, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, unidirectional: false, unidirectional_reverse: false, exit_on_eof: true, oneshot: false, unlink_unix_socket: false, exec_args: ["-c", "websocat -v ${WEBSOCAT_URI:1}"], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [], custom_reply_headers: [], websocket_version: None, websocket_dont_close: false, one_message: false, no_auto_linemode: false, buffer_size: 65536, broadcast_queue_len: 16, read_debt_handling: Warn, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: true, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, socks_destination: None, auto_socks5: None, socks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None }
[INFO  websocat::net_peer] Incoming TCP connection from Some(V4(<SomeIp>:56117))
[INFO  websocat::sessionserve] Serving 1 ongoing connections
[INFO  websocat::ws_server_peer] Incoming connection to websocket: /wss://stream.binance.com:9443/ws/btcusdt@ticker
[INFO  websocat::ws_server_peer] Upgraded
websocat: WebSocketError: I/O failure

On client host:

websocat ws://<serverIp>:8080/wss://stream.binance.com:9443/ws/btcusdt@ticker
websocat: WebSocketError: I/O failure
websocat: error running

Another go on the client, without interrupting the proxy (so after last I/O failure message), client shows the same 2 lines of errors. The proxy shows this (type 2 proxy output):

[INFO  websocat::net_peer] Incoming TCP connection from Some(V4(<myIp>:48028))
[INFO  websocat::sessionserve] Serving 1 ongoing connections
[INFO  websocat::ws_server_peer] Incoming connection to websocket: /wss://stream.binance.com:9443/ws/btcusdt@ticker
[INFO  websocat::ws_server_peer] Upgraded
[INFO  websocat::lints] Auto-inserting the line mode
[INFO  websocat::sessionserve] Serving Line2Message(Stdio) to Message2Line(WsClient("wss://stream.binance.com:9443/ws/btcusdt@ticker")) with Options { websocket_text_mode: true, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, unidirectional: false, unidirectional_reverse: false, exit_on_eof: false, oneshot: false, unlink_unix_socket: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [],custom_reply_headers: [], websocket_version: None, websocket_dont_close: false,one_message: false, no_auto_linemode: false, buffer_size: 65536,broadcast_queue_len: 16,read_debt_handling: Warn, linemode_zero_terminated: false, restrict_uri: None, serve_static_files: [], exec_set_env: false, reuser_send_zero_msg_on_disconnect: false, process_zero_sighup: false, process_exit_sighup: false, socks_destination: None, auto_socks5: None, socks5_bind_script: None, tls_domain: None, tls_insecure: false, headers_to_env: [], max_parallel_conns: None, ws_ping_interval: None, ws_ping_timeout: None }
[INFO  websocat::stdio_peer] get_stdio_peer (async)
[INFO  websocat::stdio_peer] Setting stdin to nonblocking mode
[INFO  websocat::stdio_peer] Setting stdout to nonblocking mode
[INFO  websocat::stdio_peer] Installing signal handler
[INFO  websocat::ws_client_peer] get_ws_client_peer
websocat: WebSocketError: I/O failure

[grepsuzette]

Spacing different tries:

INFO  websocat::net_peer] Incoming TCP connection from Some(V4(<someIp>:42641))
[INFO  websocat::sessionserve] Serving 1 ongoing connections
[INFO  websocat::ws_server_peer] Incoming connection to websocket: /wss://stream.binance.com:9443/ws/btcusdt@ticker
[INFO  websocat::ws_server_peer] Upgraded
websocat: WebSocketError: I/O failure
[INFO  websocat::net_peer] Incoming TCP connection from Some(V4(<someIp>:23285))
[INFO  websocat::sessionserve] Serving 1 ongoing connections
[INFO  websocat::ws_server_peer] Incoming connection to websocket: /wss://stream.binance.com:9443/ws/btcusdt@ticker
websocat: WebSocketError: I/O failure

Seems it can also fail before the upgrade to websocket (but it seems quite unusual). I think it hates me. Running websocat wss://stream.binance.com:9443/ws/btcusdt@ticker from proxy server always works.

Was thinking it could be the FW tampering with the connection. But running a VPN on the connection from client to proxy server also outputs this on the server:

[INFO  websocat::net_peer] Incoming TCP connection from Some(V4(ip:53131))
[INFO  websocat::sessionserve] Serving 1 ongoing connections
[INFO  websocat::ws_server_peer] Incoming connection to websocket: /wss://stream.binance.com:9443/ws/btcusdt@ticker
[INFO  websocat::ws_server_peer] Upgraded
websocat: WebSocketError: I/O failure
[INFO  websocat::net_peer] Incoming TCP connection from Some(V4(ip:28249))
[INFO  websocat::sessionserve] Serving 1 ongoing connections
[INFO  websocat::ws_server_peer] Incoming connection to websocket: /wss://stream.binance.com:9443/ws/btcusdt@ticker
[INFO  websocat::ws_server_peer] Upgraded
websocat: WebSocketError: I/O failure
[INFO  websocat::net_peer] Incoming TCP connection from None
[INFO  websocat::sessionserve] Serving 1 ongoing connections
[INFO  websocat::ws_server_peer] Incoming connection to websocket: /wss://stream.binance.com:9443/ws/btcusdt@ticker
websocat: WebSocketError: I/O failure

Where can it fail...

[grepsuzette]

Another info. If, like you, I run the client and the proxy on the same host, it works.

1. From local machine (using a VPN), listening/connecting to localhost:8080 to access binance, OK.
2. On proxy machine, listening/connecting to localhost:8080 to access binance, OK.

Ping time from local to the proxy server ~50ms. Binance not pingable, but time curl binance.com gives about 190ms. Could be some synchronization problem.

   local   e.websocat   i.websocat   target
     |           |            |         |
     |<----------|<-----------|<--------|

Should be very quick from internal to external websocat, but maybe e.websocat->local is slowish or something like that.

But as far as I can see it's not due to the rythm of the packets, even a slower stream such as wss://stream.binance.com:9443/ws/paxtusd@ticker gives the same problem.

[vi]

What if run both the client and proxy on the client's host as opposed to proxy's host?

Is it only failing for stream.binance.com or also for echo.websocket.org?

[vi]

Another debugging method: inspecting with socat (with ws-c:sh-c: and --ws-c-uri).

Also you can try forwarding TCP port over WebSocket though the proxy (with --binary instead of --text obviously). Then connect to 127.0.0.1:9443, sending stream.binance.com as SNI and Host:.

Also it may be worth running it with URL locked instead of specifyable by client.

Proxy may be also ran with strace -t -fo log.txt ... to produce long syscall dump, which can be inspected for problems.

[grepsuzette]

What if run both the client and proxy on the client's host as opposed to proxy's host?

This is case 1) described above. It works flawlessly. Case 2) (both on proxy host) works flawlessly too.

Is it only failing for stream.binance.com or also for echo.websocket.org?

Only for binance. echo.websocket.org and another stream (bitfinex, described above) work. I should remark that both echo.websocket.org and bitfinex streams will wait for you to send anything (the first will echo back, and bitfinex will not send anything until you subscribe to one channel sending some json. the binance stream is different, as it directly sends something to you.

(thinking i could be the problem, I tried websocat -v -Ee --text ws-l:0.0.0.0:8080 sh-c:'let i=0; while true; do let i++; echo $i; sleep 2; done' on the proxy. But it causes no problem, whatever the sleep value (tried sleep 2, 1, 0.5, 0.1, 0.01, 0.8, it always worked, with longer lines line 500 characters also works whatever the sleep value)).

Also it may be worth running it with URL locked instead of specifyable by client.

Yes. websocat -v -Ee --text ws-l:0.0.0.0:8080 exec:/bin/bash --exec-args -c 'websocat wss://stream.binance.com:9443/ws/btcusdt@ticker' on proxy fails the same.

[grepsuzette]

OK I think I understand. Your program is actually working fine.

Inspecting with strace on the outer websocat, I now get a strong feeling the connection from client to proxy is simply killed because of a connection reset (likely because it contains a forbidden url).

2869  18:17:09 socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 6
2869  18:17:09 connect(6, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 16) = 0
2869  18:17:09 gettimeofday({1565950629, 683744}, NULL) = 0
2869  18:17:09 poll([{fd=6, events=POLLOUT}], 1, 0) = 1 ([{fd=6, revents=POLLOUT}])
2869  18:17:09 sendmmsg(6, {{{msg_name(0)=NULL, msg_iov(1)=[{"\331\214\1\0\0\1\0\0\0\0\0\0\6stream\7binance\3com\0"..., 36}], msg_controllen=0, msg_flags=MSG_OOB}, 36}, {{msg_name(0)=NULL, msg_iov(1)=[{")\262\1\0\0\1\0\0\0\0\0\0\6stream\7binance\3com\0"..., 36}], msg_controllen=0, msg_flags=0}, 36}}, 2, MSG_NOSIGNAL) = 2
2869  18:17:09 poll([{fd=6, events=POLLIN}], 1, 5000) = 1 ([{fd=6, revents=POLLIN}])
2869  18:17:09 ioctl(6, FIONREAD, [241]) = 0
2869  18:17:09 recvfrom(6, "\331\214\201\200\0\1\0\t\0\0\0\0\6stream\7binance\3com\0"..., 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")},[16]) = 241
2869  18:17:09 gettimeofday({1565950629, 698071}, NULL) = 0
2869  18:17:09 poll([{fd=6, events=POLLIN}], 1, 4985) = 1 ([{fd=6, revents=POLLIN}])
2869  18:17:09 ioctl(6, FIONREAD, [197]) = 0
2869  18:17:09 recvfrom(6, ")\262\201\200\0\1\0\1\0\1\0\0\6stream\7binance\3com\0"..., 65536, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, [16]) = 197
2869  18:17:09 close(6)                 = 0
2869  18:17:09 open("/etc/gai.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
2869  18:17:09 futex(0x7f3fbe8ec3e0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
2869  18:17:09 socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE <unfinished ...>
2862  18:17:09 <... epoll_wait resumed> [{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=4194305, u64=4194305}}], 1024, -1) = 1
2862  18:17:09 clock_gettime(CLOCK_MONOTONIC, {11746908, 495603352}) = 0
2862  18:17:09 recvfrom(7, 0x7f56315f65b2, 7998, 0, NULL, NULL) = -1 ECONNRESET (Connection reset by peer)
2862  18:17:09 kill(2866, SIGKILL <unfinished ...>
2866  18:17:09 +++ killed by SIGKILL +++
2862  18:17:09 <... kill resumed> )     = 0
2862  18:17:09 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=2866, si_uid=0, si_status=SIGKILL, si_utime=0, si_stime=0} ---
2862  18:17:09 wait4(2866, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], WNOHANG, NULL) = 2866

The ECONNRESET is typical enough. The outer websocat detects a connection reset and thus produces an I/O error message. The other end on the client does the same (so same I/O error message, even though both didn't have a chance to communicate beyond the upgrade messaget). It does not come from websocat but from routers along the road (because of a plain forbidden url in the initial request) ... Probably needs some scrambling or whatever, this must be quite trivial with sh-c:. Or maybe even using wss from client to proxy.

So by now it must be safe saying above solution works. I will reopen if proved otherwise but it would surprise me.

Thanks again, man and also for the time you spent helping me. I'm very impressed with this tool and will come back if I have any other questions, which is quite probable. This is quite an amazing program you wrote :)