[question] Trying to get last price stream from Okex

[nagualcode]

Hello, I like the idea to use websocat to get last price streams from cryto exchanges, from the command line. With Binacne exchange, it is as easy as: websocat "wss://fstream.binance.com/stream?streams=$pair@markPrice" But am I having a hard time from Okex exchange. According to the docs at: https://www.okex.com/docs/en/#spot-singleness The URL is: wss://real.okex.com:8443/ws/v3 I have tried many URLs and all I can get are scramble chinese characters. Also, it seem to be DEFALTE compressed. Wonder if would be possible to get the BTC-USD last price ticker, just like it is possible with the Binance example above.

[vi]

Websocat may not be the best tool for this, unless a special deflate: overlay is added to it.

Currently it is cumbersome to extract meaningful reply from the server.

$ { printf "\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00"; echo 'QQQ' | websocat --no-line -t - wss://real.okex.com:8443/ws/v3; } | zcat

{"event":"error","message":"Unrecognized request: QQQ\n","errorCode":30039}
gzip: stdin: unexpected end of file

Websocat outputs binary WebSocket messages to stdout as is, without any separators and prefixes. This makes it able to only reliably receive one compressed message.

---

Are compressed WebSocket messages a popular thing? Are they also used in other places? I'm not sure about adding site-specific things to Websocat.

Maybe I'll add a slow "filterer" overlay that would be able to launch a process for each message that would transform it in arbitrary way.

[vi]

Another more universal idea: implement a --base64 option to encode each binary WebSocket message as base64 while preserving text messages. This should be rather universal and would allow to uncompress the messages with the next program in pipeline.

[vi]

Implemented support of base64-encoding binary WebSocket messages.

target/debug/websocat wss://real.okex.com:8443/ws/v3 --base64 | xargs -n1 -- sh -c '{ echo H4sIAAAAAAAAAA== | base64 -d; echo $0 | base64 -d; } | gunzip 2> /dev/null; echo'

qwerwqer
{"event":"error","message":"Unrecognized request: qwerwqer\n","errorCode":30039}
{"op":"subscribe","args":["Lol"]}
{"event":"error","message":"Channel Lol doesn't exist","errorCode":30040}

xargs part should obviously be rewritten to something more optimal if you want to use it for real.

[nagualcode]

Nice! Is this example of yours, where do I enter this msg: {"op": "subscribe", "args":["swap/ticker:BTC-USD-SWAP"]}

target/debug/websocat wss://real.okex.com:8443/ws/v3 --base64 | xargs -n1 -- sh -c '{ echo '{"op": "subscribe", "args":["swap/ticker:BTC-USD-SWAP"]}' | base64 -d; echo $0 | base64 -d; } | gunzip 2> /dev/null; echo' ?

[vi]

To stdin:

$ target/debug/websocat wss://real.okex.com:8443/ws/v3 --base64 | xargs -n1 -- sh -c '{ echo H4sIAAAAAAAAAA== | base64 -d;
echo $0 | base64 -d; } | gunzip 2> /dev/null; echo'
{"op": "subscribe", "args":["swap/ticker:BTC-USD-SWAP"]}
{"event":"subscribe","channel":"swap/ticker:BTC-USD-SWAP"}
{"table":"swap/ticker","data":[{"last":"7355.3","open_24h":"7134.8","best_bid":"7355.2","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536696","volume_token_24h":"62179.1352","best_ask":"7355.3","open_interest":"850830","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:46.608Z","best_bid_size":"762","best_ask_size":"4","last_qty":"2"}]}
{"table":"swap/ticker","data":[{"last":"7356.1","open_24h":"7134.8","best_bid":"7356.1","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536718","volume_token_24h":"62179.4341","best_ask":"7356.2","open_interest":"850826","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:53.163Z","best_bid_size":"522","best_ask_size":"1","last_qty":"19"}]}
{"table":"swap/ticker","data":[{"last":"7356.1","open_24h":"7134.8","best_bid":"7356.1","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536718","volume_token_24h":"62179.4341","best_ask":"7356.2","open_interest":"850826","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:53.168Z","best_bid_size":"522","best_ask_size":"1","last_qty":"3"}]}
{"table":"swap/ticker","data":[{"last":"7356.2","open_24h":"7134.8","best_bid":"7356.1","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536721","volume_token_24h":"62179.4746","best_ask":"7356.2","open_interest":"850826","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:57.477Z","best_bid_size":"522","best_ask_size":"1","last_qty":"1"}]}
{"table":"swap/ticker","data":[{"last":"7356.2","open_24h":"7134.8","best_bid":"7356.1","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536721","volume_token_24h":"62179.4746","best_ask":"7356.2","open_interest":"850804","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:57.481Z","best_bid_size":"810","best_ask_size":"3","last_qty":"1"}]}
{"table":"swap/ticker","data":[{"last":"7356.4","open_24h":"7134.8","best_bid":"7356.1","high_24h":"7432.3","low_24h":"7133.7","volume_24h":"4536722","volume_token_24h":"62179.4881","best_ask":"7356.2","open_interest":"850804","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:26:59.692Z","best_bid_size":"810","best_ask_size":"3","last_qty":"1"}]}
^C

Here is also optimized Perl-based version of the decompressor:

$ echo '{"op": "subscribe", "args":["swap/ticker:BTC-USD-SWAP"]}' | target/debug/websocat -n wss://real.okex.com:8443/ws/v3
 --base64 |  perl -wne 'use strict; use MIME::Base64; use IO::Uncompress::RawInflate qw(rawinflate);  my $b = decode_base64($_); open B, "<", \$b; my $
o=""; open C, ">", \$o; rawinflate(*B,*C); print "$o\n"'
{"event":"subscribe","channel":"swap/ticker:BTC-USD-SWAP"}
{"table":"swap/ticker","data":[{"last":"7330","open_24h":"7201","best_bid":"7331.9","high_24h":"7432.3","low_24h":"7135","volume_24h":"4524856","volume_token_24h":"61997.7287","best_ask":"7332","open_interest":"837833","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:54:05.005Z","best_bid_size":"40","best_ask_size":"1","last_qty":"0"}]}
{"table":"swap/ticker","data":[{"last":"7332","open_24h":"7201","best_bid":"7331.9","high_24h":"7432.3","low_24h":"7135","volume_24h":"4524887","volume_token_24h":"61998.1512","best_ask":"7332","open_interest":"837833","instrument_id":"BTC-USD-SWAP","timestamp":"2020-04-08T21:54:19.690Z","best_bid_size":"161","best_ask_size":"31","last_qty":"10"}]}
^C

[aldanor]

Would it be possible to add an optional deflate layer? (since there's many gzipped websocket servers these days, like okex, huobi etc)

[vi]

Note that Websocket protocol itself provides permessage-deflate compression (not supported by Websocat v1 or v3 unfortunately). Why do those servers compress data within WebSocket messages instead of using compression protocol extension? Do all of them use gzip compression (and not, for example, lzma or brotli or zstd or lz4), so that just one additional overlay enables access to multiple services?

[aldanor]

@vi I'm not sure what's the answer to the "why" question, but it seems to be growing more and more popular for whatever reason. And yes, pretty much all of them use gzip and not any other compression method - so having a simple gunzip: layer would cover all of those cases and would be immensely useful.

(if deflate layer was implemented, it would also allow you to combine it with timestamp: layer, IIUC, so you could have a stream of deflated and timestamped messages)

[aldanor]

A possible answer to "why" could be that not all libraries and clients (only a selected subset) will support permessage-deflate. Meanwhile, you can always use any websocket library and just unpack things manually using whatever tooling you want to use.

[vi]

Is there a public service supplying such "gzipped" WebSocket messages to test implementations?

wss://real.okex.com:8443/ws/v3 no longer works.

[aldanor]

This should work:

(echo -e '{"op":"subscribe","args":["spot/ticker:ETH-USDT"]}' && cat) \
  | websocat wss://real.okcoin.com:8443/ws/v3 --base64 \
  | xargs -n1 -- sh -c '{ echo H4sIAAAAAAAAAA== | base64 -d; echo $0 | base64 -d; } | gunzip 2> /dev/null; echo'

This should also work:

(echo -e '{"sub":"market.btcusdt.bbo","id":"1"}' && cat) \
  | websocat --base64 wss://api.huobi.pro/ws \
  | xargs -n1 -- sh -c '{ echo $0 | base64 -d; } | gunzip; echo'

[aldanor]

By the way, having to send a single "subscribe" message (or a few of them) is also extremely common, wonder if it's something that could be integrated in so you wouldn't have to (echo -e foo && cat) | websocat? (which is extremely unobvious if you're not aware of sending EOF and the stream closing)

E.g. websocat --send-text 'foo' --send-text 'bar' to simply have it send a few messages on connection.

[vi]

Specifying init messages on command line, to send them to WebSocket before reading further messages from stdin (unless -u) does indeed look like a reasonable feature.

[vi]

Implemented --preamble (-p) / --preamble-reverse (-P) options, which work like --send-text above.

[vi]

Implemented decompression as well.

$ websocat -nU --max-messages-rev=3  wss://real.okcoin.com:8443/ws/v3 --uncompress-deflate   -p '{"op":"subscribe","args":["spot/ticker:ETH-USDT"]}'
{"event":"subscribe","channel":"spot/ticker:ETH-USDT"}
{"table":"spot/ticker","data":[{"last":"1328.45","open_24h":"1305.39","best_bid":"1326.07","high_24h":"1339.53","low_24h":"1263.63","open_utc0":"1328.22","open_utc8":"1288.69","base_volume_24h":"1048.726528","quote_volume_24h":"1369725.083642","best_ask":"1327.39","instrument_id":"ETH-USDT","timestamp":"2022-09-24T10:32:03.130Z","best_bid_size":"0.75","best_ask_size":"0.018834","last_qty":"2.634"}]}
{"table":"spot/ticker","data":[{"last":"1328.45","open_24h":"1305.39","best_bid":"1325.03","high_24h":"1339.53","low_24h":"1263.63","open_utc0":"1328.22","open_utc8":"1288.69","base_volume_24h":"1048.726528","quote_volume_24h":"1369725.083642","best_ask":"1325.87","instrument_id":"ETH-USDT","timestamp":"2022-09-24T10:33:03.074Z","best_bid_size":"0.6281","best_ask_size":"0.018857","last_qty":"2.634"}]}

$ websocat -nU --max-messages-rev=3  wss://api.huobi.pro/ws  --uncompress-gzip -p '{"sub":"market.btcusdt.bbo","id":"1"}'
{"id":"1","status":"ok","subbed":"market.btcusdt.bbo","ts":1664015732450}
{"ping":1664015732924}
{"ch":"market.btcusdt.bbo","ts":1664015733639,"tick":{"seqId":159767128361,"ask":19035.2,"askSize":0.11906853906499539,"bid":19035.19,"bidSize":0.38414,"quoteTime":1664015733638,"symbol":"btcusdt"}}

[aldanor]

@vi Looks awesome, thanks!