Default compression settings of 10.0 trigger NegotiationError when connecting to buggy servers like bybit

[daesungkim95]

I'm trying to use a websocket api from bybit, which you can find the manual in the link below.

https://bybit-exchange.github.io/docs/linear/#t-websocket

When used with "websockets 10.0", it raises exception when connecting to the websocket server, but I solved this by giving the argument "compression=None".

However, even with "websockets 9.1", I cannot receive any data from this websocket server. I do not have any knowledge on websocket protocol, but when I used other library such as "websocket-client", or different languace, C#, I could receive data successfully. So, I suspect that this problem occurs because of the "websockets" library not by the bybit websocket server.

Below I add my code.

import asyncio
import websockets
import json

url_bybit = 'wss://stream.bybit.com/realtime_public'

async def get_bybit_price():
    ws_bybit = await websockets.connect(url_bybit, compression=None)

    data = {
        "op": "subscribe",
        "args": ["candle.1.BTCUSDT"]
    }
    await ws_bybit.send(json.dumps(data))

    while True:
        data = await ws_bybit.recv()
        data = data.replace("'", '"')
        data = json.loads(data)

        print(data)

asyncio.run(get_bybit_price())

Edit: I found that this code works with "websockets 9.0"

[aaugustin]

I suspect that this problem occurs because of the "websockets" library not by the bybit websocket server.

Your suspicion is wrong.

The server doesn't implement https://www.rfc-editor.org/rfc/rfc7692.html#section-7.1 correctly.

Could you please report it to bybit.com and add a link here, so that other affected users can figure it out easily?

Specifically, the server breaks this requirement:

A server accepts an extension negotiation offer with this parameter by including the "server_max_window_bits" extension parameter in the extension negotiation response to send back to the client with the same or smaller value as the offer.

Note how there's no server_max_window_bits parameter in the Sec-WebSocket-Extensions header in the response, despite the Sec-WebSocket-Extensions header in the request containing this parameter:

DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:websockets.client:= connection is CONNECTING
DEBUG:websockets.client:> GET /realtime_public HTTP/1.1
DEBUG:websockets.client:> Host: stream.bybit.com
DEBUG:websockets.client:> Upgrade: websocket
DEBUG:websockets.client:> Connection: Upgrade
DEBUG:websockets.client:> Sec-WebSocket-Key: zre7Cvol9I8nk0KKe9UyqA==
DEBUG:websockets.client:> Sec-WebSocket-Version: 13
DEBUG:websockets.client:> Sec-WebSocket-Extensions: permessage-deflate; server_max_window_bits=12; client_max_window_bits=12
DEBUG:websockets.client:> User-Agent: Python/3.10 websockets/10.0
DEBUG:websockets.client:< HTTP/1.1 101 Switching Protocols
DEBUG:websockets.client:< Connection: upgrade
DEBUG:websockets.client:< Date: Tue, 12 Oct 2021 06:55:54 GMT
DEBUG:websockets.client:< Upgrade: websocket
DEBUG:websockets.client:< Sec-WebSocket-Accept: vyj9cBmZdB+VOCShw0sTY7zgqj8=
DEBUG:websockets.client:< Sec-WebSocket-Extensions: permessage-deflate; server_no_context_takeover; client_no_context_takeover
DEBUG:websockets.client:< X-Cache: Miss from cloudfront
DEBUG:websockets.client:< Via: 1.1 16de6e3636993b2d3f832b9ae653bd69.cloudfront.net (CloudFront)
DEBUG:websockets.client:< X-Amz-Cf-Pop: CDG50-P1
DEBUG:websockets.client:< X-Amz-Cf-Id: DZcGZcQRgLbvu9uHCyM2Nzy0kyBof8DzFbfTfe4me1RUPPsU_c3CxA==
DEBUG:websockets.client:! failing connection with code 1006
DEBUG:websockets.client:x closing TCP connection
DEBUG:websockets.client:= connection is CLOSED
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/myk/.pyenv/versions/3.10.0/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/myk/.pyenv/versions/3.10.0/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "<stdin>", line 2, in test
  File "/Users/myk/dev/websockets/src/websockets/legacy/client.py", line 650, in __await_impl_timeout__
    return await asyncio.wait_for(self.__await_impl__(), self.open_timeout)
  File "/Users/myk/.pyenv/versions/3.10.0/lib/python3.10/asyncio/tasks.py", line 447, in wait_for
    return fut.result()
  File "/Users/myk/dev/websockets/src/websockets/legacy/client.py", line 659, in __await_impl__
    await protocol.handshake(
  File "/Users/myk/dev/websockets/src/websockets/legacy/client.py", line 332, in handshake
    self.extensions = self.process_extensions(
  File "/Users/myk/dev/websockets/src/websockets/legacy/client.py", line 221, in process_extensions
    raise NegotiationError(
websockets.exceptions.NegotiationError: Unsupported extension: name = permessage-deflate, params = [('server_no_context_takeover', None), ('client_no_context_takeover', None)]

[daesungkim95]

Thanks for your careful review. I will forward your suggestion to the Bybit team. Can you please check two more servers? They both work well with "websockets 9.1", but I cannot receive any data from both of them with "websockets 10.0".

import asyncio
import websockets
import uuid
import json

url_binance = 'wss://fstream.binance.com/ws/'
url_upbit = 'wss://api.upbit.com/websocket/v1'

async def get_binance_price():
    ws_binance = await websockets.connect(url_binance)

    data = {
        "method": "SUBSCRIBE",
        "params": ['btcusdt@kline_1m']
    }
    await ws_binance.send(json.dumps(data))

    while True:
        data = await ws_binance.recv()
        data = data.replace("'", '"')
        data = json.loads(data)

        print(data)

async def get_upbit_price():
    ws_upbit = await websockets.connect(url_upbit)

    data = [
        {"ticket": str(uuid.uuid4())[:6]},
        {"type": "trade", "codes": ["KRW-BTC"]},
        {"format": "SIMPLE"}
    ]
    await ws_upbit.send(json.dumps(data))

    while True:
        data = await ws_upbit.recv()
        data = data.decode('utf8').replace("'", '"')
        data = json.loads(data)

        print(data)

asyncio.run(get_binance_price())
#asyncio.run(get_upbit_price())

[aaugustin]

If you can establish a connection but not receive data, there's a high chance it's #1065.

[aaugustin]

I'm planning to do something about this, because clearly websockets assumes too much about the correctness of the servers it connects to, but not sure what exactly.

[daesungkim95]

I checked that the additional two cases could also be resolved by providing 'compression=None' option when connecting.

As a result, all of my problems could be solved with 'compression=None' option.

I do not know anything about the websockets, but changing the default value of compression to None could be an option.

Thanks for your help!

[mick-phemex]

I just met the same problem, but I found out the best solution is to downgrade the version of the package to 8.1. Then problem solved. I strongly suggest the author to check the modification between the 8.1 ver. and the other latest ver.

[aaugustin]

Version 10.1 is available on PyPI.