Websocket connection error

[Pusherman2]

Hi there,

Very nice project. I am running the cryptofeed strategy with some modifications but I notice that the websocket connection sometimes hangs/freezes and no new liquidations are coming in via the feed. Please find the exact error logs below:

2022-09-19 22:17:05,570 :: ERROR :: ftx_algotrading :: Future exception was never retrieved future: <Future finished exception=ConnectionClosedOK(Close(code=1001, reason="bytearray(b'CloudFlare WebSocket proxy restarting')"), Close(code=1001, reason="bytearray(b'CloudFlare WebSocket proxy restarting')"), True)> websockets.exceptions.ConnectionClosedOK: received 1001 (going away) bytearray(b'CloudFlare WebSocket proxy restarting'); then sent 1001 (going away) bytearray(b'CloudFlare WebSocket proxy restarting')

('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) /!\ FTX REST API EXCEPTION: Please retry request /!\ FTX REST API EXCEPTION: Please retry request /!\ FTX REST API EXCEPTION: Please retry request /!\ FTX REST API EXCEPTION: Please retry request /!\ FTX REST API EXCEPTION: Please retry request 2022-09-19 15:07:21,653 : WARNING : BINANCE_FUTURES.ws.3: received no messages within timeout, restarting connection 2022-09-19 15:23:03,161 : WARNING : BINANCE_FUTURES.ws.3: received no messages within timeout, restarting connection ^C2022-09-19 16:58:35,880 : ERROR : BINANCE_FUTURES.http.2.http.2: connection closed in read()

Sometimes the websocket/API return the 1006 error.

Do you maybe have an idea how to succesfully restart the ws connection?

Kind regards,

Pusherman

[Pusherman2]

Some additional error logs: `2022-09-26 13:55:03,512 : WARNING : BINANCE_FUTURES.ws.4: received no messages within timeout, restarting connection /!\ FTX REST API EXCEPTION: Please retry request /!\ FTX REST API EXCEPTION: Please retry request ^C^C^CTraceback (most recent call last): File "/home/crackbox/.local/lib/python3.9/site-packages/cryptofeed/feedhandler.py", line 151, in run loop.run_forever() File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run File "uvloop/handles/poll.pyx", line 213, in uvloop.loop.__on_uvpoll_event File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run File "uvloop/cbhandles.pyx", line 66, in uvloop.loop.Handle._run File "uvloop/loop.pyx", line 397, in uvloop.loop.Loop._read_from_self File "uvloop/loop.pyx", line 402, in uvloop.loop.Loop._invoke_signals File "uvloop/loop.pyx", line 377, in uvloop.loop.Loop._ceval_process_signals KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/crackbox/Documents/ftx_algotrading3/ftx_algotrading.py", line 24, in strategy.run() File "/home/crackbox/Documents/ftx_algotrading3/strategies/cryptofeed_strategy/cryptofeed_strategy.py", line 600, in run CryptofeedService.start_cryptofeed() File "/home/crackbox/Documents/ftx_algotrading3/strategies/cryptofeed_strategy/cryptofeed_service.py", line 84, in start_cryptofeed f.run(install_signal_handlers=False) File "/home/crackbox/.local/lib/python3.9/site-packages/cryptofeed/feedhandler.py", line 157, in run self.stop(loop=loop) File "/home/crackbox/.local/lib/python3.9/site-packages/cryptofeed/feedhandler.py", line 195, in stop loop.run_until_complete(asyncio.gather(*shutdown_tasks)) File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run File "uvloop/handles/poll.pyx", line 213, in uvloop.loop.__on_uvpoll_event File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run File "uvloop/cbhandles.pyx", line 66, in uvloop.loop.Handle._run File "uvloop/loop.pyx", line 397, in uvloop.loop.Loop._read_from_self File "uvloop/loop.pyx", line 402, in uvloop.loop.Loop._invoke_signals File "uvloop/loop.pyx", line 377, in uvloop.loop.Loop._ceval_process_signals KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/crackbox/Documents/ftx_algotrading3/ftx_algotrading.py", line 26, in strategy.cleanup() File "/home/crackbox/Documents/ftx_algotrading3/strategies/cryptofeed_strategy/cryptofeed_strategy.py", line 144, in cleanup self._t.join() File "/usr/local/lib/python3.9/threading.py", line 1053, in join self._wait_for_tstate_lock() File "/usr/local/lib/python3.9/threading.py", line 1069, in _wait_for_tstate_lock elif lock.acquire(block, timeout): KeyboardInterrupt ^CException ignored in: <module 'threading' from '/usr/local/lib/python3.9/threading.py'> Traceback (most recent call last): File "/usr/local/lib/python3.9/threading.py", line 1448, in _shutdown lock.acquire() KeyboardInterrupt: `

`('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) 2022-09-26 18:05:47,650 : ERROR : BINANCE_FUTURES.http.3.http.3: encountered an exception, reconnecting in 1.0 seconds Traceback (most recent call last): File "/home/crackbox/.local/lib/python3.9/site-packages/cryptofeed/connection_handler.py", line 69, in _create_connection await self._handler(connection, self.handler) File "/home/crackbox/.local/lib/python3.9/site-packages/cryptofeed/connection_handler.py", line 115, in _handler async for message in connection.read(): File "/home/crackbox/.local/lib/python3.9/site-packages/cryptofeed/connection.py", line 252, in read yield await self._read_address(addr, header) File "/home/crackbox/.local/lib/python3.9/site-packages/cryptofeed/connection.py", line 244, in _read_address response.raise_for_status() File "/home/crackbox/.local/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status raise ClientResponseError( aiohttp.client_exceptions.ClientResponseError: 418, message='Unknown', url=URL('https://fapi.binance.com/fapi/v1/openInterest?symbol=BATUSDT') 2022-09-26 18:11:13,614 : WARNING : BINANCE_FUTURES.ws.4: received no messages within timeout, restarting connection

^C^C^CTraceback (most recent call last): File "/home/crackbox/.local/lib/python3.9/site-packages/cryptofeed/feedhandler.py", line 151, in run loop.run_forever() File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run File "uvloop/handles/poll.pyx", line 213, in uvloop.loop.__on_uvpoll_event File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run File "uvloop/cbhandles.pyx", line 66, in uvloop.loop.Handle._run File "uvloop/loop.pyx", line 397, in uvloop.loop.Loop._read_from_self File "uvloop/loop.pyx", line 402, in uvloop.loop.Loop._invoke_signals File "uvloop/loop.pyx", line 377, in uvloop.loop.Loop._ceval_process_signals KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/crackbox/Documents/ftx_algotrading2/ftx_algotrading.py", line 24, in strategy.run() File "/home/crackbox/Documents/ftx_algotrading2/strategies/cryptofeed_strategy/cryptofeed_strategy.py", line 605, in run CryptofeedService.start_cryptofeed() File "/home/crackbox/Documents/ftx_algotrading2/strategies/cryptofeed_strategy/cryptofeed_service.py", line 84, in start_cryptofeed f.run(install_signal_handlers=False) File "/home/crackbox/.local/lib/python3.9/site-packages/cryptofeed/feedhandler.py", line 157, in run self.stop(loop=loop) File "/home/crackbox/.local/lib/python3.9/site-packages/cryptofeed/feedhandler.py", line 195, in stop loop.run_until_complete(asyncio.gather(*shutdown_tasks)) File "uvloop/loop.pyx", line 1511, in uvloop.loop.Loop.run_until_complete File "uvloop/loop.pyx", line 1504, in uvloop.loop.Loop.run_until_complete File "uvloop/loop.pyx", line 1377, in uvloop.loop.Loop.run_forever File "uvloop/loop.pyx", line 555, in uvloop.loop.Loop._run File "uvloop/handles/poll.pyx", line 213, in uvloop.loop.__on_uvpoll_event File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run File "uvloop/cbhandles.pyx", line 66, in uvloop.loop.Handle._run File "uvloop/loop.pyx", line 397, in uvloop.loop.Loop._read_from_self File "uvloop/loop.pyx", line 402, in uvloop.loop.Loop._invoke_signals File "uvloop/loop.pyx", line 377, in uvloop.loop.Loop._ceval_process_signals KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/crackbox/Documents/ftx_algotrading2/ftx_algotrading.py", line 26, in strategy.cleanup() File "/home/crackbox/Documents/ftx_algotrading2/strategies/cryptofeed_strategy/cryptofeed_strategy.py", line 142, in cleanup self._t.join() File "/usr/local/lib/python3.9/threading.py", line 1053, in join self._wait_for_tstate_lock() File "/usr/local/lib/python3.9/threading.py", line 1069, in _wait_for_tstate_lock elif lock.acquire(block, timeout): KeyboardInterrupt ^CException ignored in: <module 'threading' from '/usr/local/lib/python3.9/threading.py'> Traceback (most recent call last): File "/usr/local/lib/python3.9/threading.py", line 1448, in _shutdown lock.acquire() KeyboardInterrupt:`

I checked but I could not find a way to handle the exception and succesfully reconnect. Perhaps it's possible to close all running/opened connections and restart from scratch?

Hope to hear your insights on this :)

Kind regards,

Pusherman

[AntoineLep]

Hi @Pusherman2,

Thanks for your interest to this project :). From what I can see with the logs you provided it's seems to be a problem with cryptofeed itself.

There is an endless reset of the cryptofeed runner in cryptofeed_service.py file

while True:
    try:
        # Configuration stuff

        print("Starting feedhandler for exchanges:", ', '.join(configured))
        f.run(install_signal_handlers=False)
    except KeyboardInterrupt:  # pragma: no cover
        raise
    except Exception as e:
        logging.error(e)
        pass

So unless we receive an explicit KeyboardInterrupt error, it should restart the runner. You can remove this code block and see if it fixes your issue:

except KeyboardInterrupt:  # pragma: no cover
    raise

The bad part of not handling KeyboardInterrupt error is that you might have to find a workaround to kill the thread :)

Let me know it it helps.

Regards,

Antoine

[Pusherman2]

Hi @AntoineLep,

Thank you for your reply, creating this repo and sharing it. It is much appreciated :)

As for the error, it is not so much the keyboard interrupts, that's just me interrupting the script from running. But I have seen similar errors in the Cryptofeed Issue page, however, I couldn't find a satisfying solution which applied. The errors are preceded by API errors being returned after which, the request is not repeated or the (websocket) connection is being reset. For example, the logger will return for example an FTX API exception where it is requesting to retry the request and the rest hangs after that. A different example is whenever the websocket gets timed out, it sometimes does not manage to come back up although I am playing around with ping intervals, timeouts, queue to prevent the websocket connection from falling flat.

I've implemented some try except blocks with error logging in order to have some more details so I can analyse them. I've also added some conditions to check if requests pass/are valid and to retry if they fail. I'm hoping this will fix some of the hangups.

I will update as I have some more debug logging.

Thank you and kind regards,

Pusherman

[Pusherman2]

Hi @AntoineLep

All is well now, adding some retries for all FTX api requests stops the script from crashing/hanging.

Will close the issue now.

Kind regards,

Pusherman