search

LUCIT-Systems-and-Development / unicorn-binance-websocket-api

A Python SDK by LUCIT to use the Binance Websocket API`s (com+testnet, com-margin+testnet, com-isolated_margin+testnet, com-futures+testnet, com-coin_futures, us, tr, dex/chain+testnet) in a simple, fast, flexible, robust and fully-featured way.
https://unicorn-binance-websocket-api.docs.lucit.tech/
Other
677 stars 166 forks source link

Websockets stop with error with python 3.8.10? #235

Closed CharlyEmpereurmot closed 2 years ago

CharlyEmpereurmot commented 2 years ago

Check this or we will delete your issue. (fill in the checkbox with an X like so: [x])

Select one:

Environment

What kind of internet connection do you have?

I'm running my code on an Oracle Free Tier hosted VPS with solid connection.

Average system load (CPU)

4 cpu with low system load (<10%), although I believe they are shared on the VPS.

Hardware specification

24 Gb RAM, enough disk space free (> 30Gb).

Operating System?

Options

Which endpoint do you connect?

binance.com

Python Version Requirement

Exact Python Version?

Python 3.8.10

Pip Version?

pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)

Dependencies

pip_list.txt

UNICORN Binance WebSocket API Version?

unicorn-binance-websocket-api 1.36.0

Description of your issue

Hi,

When running the websockets stream I get this message in my logs:

BinanceWebSocketApiSocket.start_socket(f728f3f2ab52-d04e-607b-65dc-9a1ccc98, ['kline_1d'], ['xrpusdt']) - Exception ConnectionClosed - error_msg: sent 1011 (unexpected error) keepalive ping timeout; no close fra
me received
BinanceWebSocketApiManager.stream_is_crashing(f728f3f2ab52-d04e-607b-65dc-9a1ccc98)
XRPUSDT_1d -- 09:36:21 on 09-02-2022 UTC -- WARNING: Stream disconnected
XRPUSDT_1d -- 09:36:31 on 09-02-2022 UTC -- WARNING: Stream connected

Not sure how much of an error this one is, it seems to recover.

But then I also have this error that pops:

Exception in thread Thread-936:
Traceback (most recent call last):
  File "/home/charlyvps/.local/lib/python3.8/site-packages/unicorn_binance_websocket_api/sockets.py", line 66, in start_socket
    async with BinanceWebSocketApiConnection(self.manager,
  File "/home/charlyvps/.local/lib/python3.8/site-packages/unicorn_binance_websocket_api/connection.py", line 132, in __aenter__
    self.manager.websocket_list[self.stream_id] = await self._conn.__aenter__()
  File "/home/charlyvps/.local/lib/python3.8/site-packages/websockets/legacy/client.py", line 633, in __aenter__
    return await self
  File "/home/charlyvps/.local/lib/python3.8/site-packages/websockets/legacy/client.py", line 650, in __await_impl_timeout__
    return await asyncio.wait_for(self.__await_impl__(), self.open_timeout)
  File "/usr/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/charlyvps/.local/lib/python3.8/site-packages/unicorn_binance_websocket_api/manager.py", line 476, in _create_stream_thread
    loop.run_until_complete(socket.start_socket())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/charlyvps/.local/lib/python3.8/site-packages/unicorn_binance_websocket_api/sockets.py", line 209, in start_socket
    self.manager.stream_is_crashing(self.stream_id, error_msg)
UnboundLocalError: local variable 'error_msg' referenced before assignment

And this one also:

BinanceWebSocketApiSocket.start_socket(f728f3f2ab52-d04e-607b-65dc-9a1ccc98, ['kline_1d'], ['xrpusdt'] socket_id=b1259da0-b9d5-4d90-8e71-9a11ea28a59b recent_socket_id=b1259da0-b9d5-4d90-8e71-9a11ea28a59b - Sending payload - exit because its not the recent socket id! stream_id=f728f3f2ab52-d04e-607b-65dc-9a1ccc98, recent_socket_id=c8306dc6-cf6d-4135-8ae0-b26829e08e26

Then the problem I am seeing is that I have stopped listening to the websokcets at some point.

Here is a minimal version of the code I'm using:

import unicorn_binance_websocket_api

def stream_data(binance_websocket_api_manager):

    while True:
        stream_payload = binance_websocket_api_manager.pop_stream_data_from_stream_buffer()
        stream_signal = binance_websocket_api_manager.pop_stream_signal_from_stream_signal_buffer()

        if stream_signal is not False:
            pair, period = streams_uuids[stream_signal['stream_id']]

            if stream_signal['type'] == 'CONNECT':
                if not data[pair]['stream_connected']:
                    data[pair]['stream_connected'] = True
                    print(f"{pair.upper()}_{period} -- {h.format_s_timestamp(datetime.now(tz=pytz.utc).timestamp())} -- WARNING: Stream connected")

            elif stream_signal['type'] == 'DISCONNECT':
                if data[pair]['stream_connected']:
                    data[pair]['stream_connected'] = False
                    print(f"{pair.upper()}_{period} -- {h.format_s_timestamp(datetime.now(tz=pytz.utc).timestamp())} -- WARNING: Stream disconnected")

        # process payload from websocket
        if stream_payload is False:
            time.sleep(0.1)
        else:
            do_stuff()

if __name__ == "__main__":
      pairs = ['xrpusdt']  # and many others, 40 pairs here
      period = '1d'
      binance_websocket_api_manager = unicorn_binance_websocket_api.BinanceWebSocketApiManager(exchange="binance.com", enable_stream_signal_buffer=True)

     for pair in pairs:
          stream_uuid = binance_websocket_api_manager.create_stream(channels=[f"kline_{period}"], markets=[pair])
          data[pair] = { 'stream_connected': False }
          streams_uuids[stream_uuid] = pair, period

    # start websockets
    worker_thread = threading.Thread(target=stream_data, args=(binance_websocket_api_manager,))
    worker_thread.start()

Can you please indicate if I am indeed seeing bugs, or if I'm just mis-using the library? Is this the correct way to stream websockets? I am using this code with 40 pairs, each using a single period.

Thank you very much

oliver-zehentleitner commented 2 years ago

your script looks good, but actually you dont need to start a thread, because your main thread is not doing something, so you can call stream_data() also directly.

as you say, the error 1011 of the connection is handled (and is not your fault), but during handling the exception an other error occurs. i fixed it now. the fix is included in the next update!

thanks for the good and complete report!

CharlyEmpereurmot commented 2 years ago

Thanks for the prompt answer and the advices! Good to see this is already fixed :)

Just to be sure ... this message in particular, this is also tackled by the recent fix? Or harmless? Can you please confirm that the stream uuid is NOT changing upon reconnection? (that's my current understanding and I rely on this behavior in my coding)

BinanceWebSocketApiSocket.start_socket(f728f3f2ab52-d04e-607b-65dc-9a1ccc98, ['kline_1d'], ['xrpusdt'] socket_id=b1259da0-b9d5-4d90-8e71-9a11ea28a59b recent_socket_id=b1259da0-b9d5-4d90-8e71-9a11ea28a59b - Sending payload - exit because its not the recent socket id! stream_id=f728f3f2ab52-d04e-607b-65dc-9a1ccc98, recent_socket_id=c8306dc6-cf6d-4135-8ae0-b26829e08e26

Thanks

oliver-zehentleitner commented 2 years ago

yes, thats fine!

when you create a stream, then you receive a stream_id which is unique and never changes.

but every stream needs a valid socket and each socket has also an unique id and if one stream has more than one active sockets, then all sockets except the newest one are closing them self. thats what happend here. one socket exists because its not the recent one...

oliver-zehentleitner commented 2 years ago

https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/releases/tag/1.36.1