search

bmoscon / cryptofeed

Cryptocurrency Exchange Websocket Data Feed Handler
Other
2.24k stars 688 forks source link

ConnectionResetError: [Errno 104] Connection Reset by peer #22

Closed eliquinox closed 6 years ago

eliquinox commented 6 years ago 
018-05-11 03:21:20,137 : WARNING : Error in data transfer
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/websockets-4.0.1-py3.6-linux-x86_64.egg/websockets/protocol.py", line 496, in transfer_data
    msg = yield from self.read_message()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/websockets-4.0.1-py3.6-linux-x86_64.egg/websockets/protocol.py", line 526, in read_message
    frame = yield from self.read_data_frame(max_size=self.max_size)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/websockets-4.0.1-py3.6-linux-x86_64.egg/websockets/protocol.py", line 591, in read_data_frame
    frame = yield from self.read_frame(max_size)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/websockets-4.0.1-py3.6-linux-x86_64.egg/websockets/protocol.py", line 632, in read_frame
    extensions=self.extensions,
  File "/home/ubuntu/.local/lib/python3.6/site-packages/websockets-4.0.1-py3.6-linux-x86_64.egg/websockets/framing.py", line 100, in read
    data = yield from reader(2)
  File "/home/ubuntu/anaconda3/lib/python3.6/asyncio/streams.py", line 663, in readexactly
    raise self._exception
  File "/home/ubuntu/anaconda3/lib/python3.6/asyncio/selector_events.py", line 723, in _read_ready
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer

16 handled the ConnectionClosed exceptions, however streaming and writing Bitmex data has caused the following error after one and half days of keeping connection alive. Some sources say that there is not much that can be done about this error: https://stackoverflow.com/questions/20568216/python-handling-socket-error-errno-104-connection-reset-by-peer?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

Was wondering if there is another try-catch block that can be defined in feedhandler.py to handle this.

bmoscon commented 6 years ago

I have a fix for this but it depends on a fix in the websockets library. Still waiting on them to release a new version.

bmoscon commented 6 years ago

@eliquinox can you tell me what version you were running? Was it installed from pypi? I pushed some changes last week to the repo, but didnt release a new version yet. Do you know if you had the latest set of changes?

bmoscon commented 6 years ago

I do see the point you are making though, its not an exception from the websockets library, so I should add another except handler for this.

bmoscon commented 6 years ago

I think my latest commit should address this. Hard to test for since the connections can take a long time to error out. I've updated my servers, let me know if you encounter any errors on your side. If I dont hear anything within a week or so I'll close this out (Unless I hit an error).

eliquinox commented 6 years ago

I tried it with your latest commit in feedhandler.py and got the above output in the logger.

eliquinox commented 6 years ago

Perhaps you can use the same recovery logic to handle ConncetionResetError as you do to recover from ConnectionClosed. When is the websockets library scheduled to push the new version?

bmoscon commented 6 years ago

I made some commits after I asked that question (and I even pushed a new version to pypi). Did you upgrade to that version before testing again? The latest version does have checks for connectionResetError

eliquinox commented 6 years ago

@bmoscon could you give a link to the specific branch / version you are referring to? I will make sure to run tests again when I upgrade. Cheers.

bmoscon commented 6 years ago

https://pypi.org/project/cryptofeed/0.10.1/

so you can pip install it (or upgrade)

Its on master, commit: 359aee907c37e0c1517a7563dd930e6a480e3834

so you could just make sure you're on master and do a git pull (if you're running from source)

bmoscon commented 6 years ago

@eliquinox were you able to confirm you're using the correct version and start a test? I've been running the latest vs bitmex for ~3 days now without issue. I'll leave it going for at least another week or so, but hoping someone else can help me confirm its working, and hopefully one of us will encounter an error that causes a reconnect

bmoscon commented 6 years ago

been running for 3+ days now - I hit an error and it successfully reconnected:

2018-05-14 10:13:03,065 : WARNING : Feed BITMEX encountered connection issue WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason - reconnecting...

bmoscon commented 6 years ago

Going to close this as I no longer have seen any of these errors. If you have issues still please re-open @eliquinox !