spesmilo / electrumx

Alternative implementation of spesmilo/electrum-server
MIT License
441 stars 353 forks source link

random crashes related to websockets connections #169

Closed cipig closed 2 years ago

cipig commented 2 years ago

I get random crashes with errors like this:

Mar 05 13:24:39 electrum3 electrumx_server[2795038]: ERROR:electrumx:ElectrumX server terminated abnormally
Mar 05 13:24:39 electrum3 electrumx_server[2795038]: Traceback (most recent call last):
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/websockets/legacy/protocol.py", line 944, in transfer_data
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     message = await self.read_message()
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/websockets/legacy/protocol.py", line 1013, in read_message
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     frame = await self.read_data_frame(max_size=self.max_size)
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/websockets/legacy/protocol.py", line 1088, in read_data_frame
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     frame = await self.read_frame(max_size)
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/websockets/legacy/protocol.py", line 1143, in read_frame
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     frame = await Frame.read(
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/websockets/legacy/framing.py", line 70, in read
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     data = await reader(2)
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/lib/python3.9/asyncio/streams.py", line 723, in readexactly
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await self._wait_for_data('readexactly')
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/lib/python3.9/asyncio/streams.py", line 517, in _wait_for_data
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await self._waiter
Mar 05 13:24:39 electrum3 electrumx_server[2795038]: asyncio.exceptions.CancelledError
Mar 05 13:24:39 electrum3 electrumx_server[2795038]: The above exception was the direct cause of the following exception:
Mar 05 13:24:39 electrum3 electrumx_server[2795038]: Traceback (most recent call last):
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/bin/electrumx_server", line 35, in main
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     asyncio.run(controller.run())
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     return loop.run_until_complete(main)
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "uvloop/loop.pyx", line 1501, in uvloop.loop.Loop.run_until_complete
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/electrumx/lib/server_base.py", line 129, in run
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await server_task
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/electrumx/lib/server_base.py", line 102, in serve
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await self.serve(shutdown_event)
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/electrumx/server/controller.py", line 134, in serve
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await group.spawn(wait_for_catchup())
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/aiorpcx/curio.py", line 297, in __aexit__
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await self.join()
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/electrumx/lib/util.py", line 370, in join
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     task.result()
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/electrumx/server/session.py", line 642, in serve
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await group.spawn(self._manage_servers())
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/aiorpcx/curio.py", line 297, in __aexit__
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await self.join()
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/electrumx/lib/util.py", line 370, in join
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     task.result()
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/electrumx/server/session.py", line 1017, in notify
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await self.send_notification('blockchain.headers.subscribe', args)
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/aiorpcx/session.py", line 545, in send_notification
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await self._send_message(message)
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/aiorpcx/session.py", line 153, in _send_message
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await self.transport.write(message)
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/aiorpcx/websocket.py", line 83, in write
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await self.websocket.send(framed_message)
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/websockets/legacy/protocol.py", line 619, in send
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     await self.ensure_open()
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:   File "/usr/local/lib/python3.9/dist-packages/websockets/legacy/protocol.py", line 929, in ensure_open
Mar 05 13:24:39 electrum3 electrumx_server[2795038]:     raise self.connection_closed_exc()
Mar 05 13:24:39 electrum3 electrumx_server[2795038]: websockets.exceptions.ConnectionClosedError: sent 1011 (unexpected error) keepalive ping timeout; no close frame received

it seems to be related to websockets connections i am using debian 11 (bullseye-backports) and latest electrumx (master branch)

SomberNight commented 2 years ago

This looks related to https://github.com/kyuupichan/electrumx/pull/1022 and https://github.com/kyuupichan/electrumx/commit/742f4914803d9fec0870996801a892594bd4ae12