Closed Kiessar closed 1 year ago
@Kiessar In your script, the pairs is not "ALGOXBT" but "ALGO/XBT". Also pairdatas is not defined so you should fix that also. Finally when you have the "socket.send() raised exception." at which time does it occurs ? Because I don't see it, maybe it's in your save method that you send something badly formatted.
@Atem18 thx for your fast reply. The missing parts are, because I tried to focus on the kraky part. For thefull code please look here: WSDatefetcher
As you may have noticed I tried to put a lot of "try:except" block in the code but I cannot make out where the issue arises. If would need to guess, I'd say it looses connection and getting the error on reconnecting.
Installed packages (if something is wrong with a version I use):
aiodns==2.0.0
aiohttp==3.7.4.post0
anyio==3.5.0
APScheduler==3.6.3
async-timeout==3.0.1
attrs==21.2.0
backcall==0.2.0
backtrader==1.9.76.123
bt-ccxt-store==1.0
ccxt==1.51.21
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
charset-normalizer==2.0.12
click==8.0.3
cryptography==3.4.7
cycler==0.10.0
deap==1.3.1
decorator==5.0.9
distlib==0.3.4
empyrical==0.5.5
filelock==3.6.0
Flask==2.0.2
greenlet==1.1.1
h11==0.12.0
httpcore==0.14.7
httpx==0.21.3
idna==2.10
ipython==7.23.1
ipython-genutils==0.2.0
itsdangerous==2.0.1
jedi==0.18.0
Jinja2==3.0.2
joblib==1.0.1
kiwisolver==1.3.1
krakenex==2.1.0
kraky==2022.1.24
lxml==4.6.3
MarkupSafe==2.0.1
matplotlib==3.2.2
matplotlib-inline==0.1.2
multidict==5.1.0
numpy==1.20.1
packaging==21.3
pandas==1.2.2
pandas-datareader==0.9.0
parso==0.8.2
patsy==0.5.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.2.0
platformdirs==2.5.1
pluggy==1.0.0
prompt-toolkit==3.0.18
ptyprocess==0.7.0
py==1.11.0
pycares==4.0.0
pycparser==2.20
pyfolio==0.9.2
Pygments==2.9.0
PyMySQL==1.0.2
pyparsing==2.4.7
python-dateutil==2.8.1
python-telegram-bot==13.5
pytrends==4.7.3
pytz==2021.1
requests==2.25.1
rfc3986==1.5.0
scikit-learn==0.24.1
scipy==1.6.2
seaborn==0.11.1
six==1.15.0
sklearn==0.0
sniffio==1.2.0
SQLAlchemy==1.4.23
statsmodels==0.12.2
stopit==1.1.2
threadpoolctl==2.1.0
toml==0.10.2
torch==1.8.1+cpu
torchaudio==0.8.1
torchvision==0.9.1+cpu
tornado==6.1
tox==3.24.5
TPOT==0.11.7
tqdm==4.60.0
traitlets==5.0.5
typer==0.4.0
typing-extensions==3.10.0.0
tzlocal==2.1
update-checker==0.18.0
urllib3==1.26.3
virtualenv==20.13.4
waitress==2.0.0
wcwidth==0.2.5
websockets==10.2
Werkzeug==2.0.2
xgboost==1.3.3
yarl==1.6.3
@Kiessar To be honest, I am not sure why you are getting this error. As you can see here: https://github.com/Atem18/kraky/blob/master/kraky/ws.py If you use my methods, I am catching the websockets errors already. I also tried already to cut my internet connection and resume and it's reconnecting, that's the principle of websockets. Even while trying your script, I tried to disconnect from the internet and reconnect and it's still working. After how many times do you have this error ?
@Atem18 I didn't go into the analysis this deep but I setup a new venv and installed only plain kraky. If this works I'm going to do a diff on the verisons and let you know. It seems that error starts between 1 and 6 hours. Because I started a new clean run, I will let you know how long it took. Thank you for your effort and assisstance.
@Kiessar no problem and let me know :)
@Atem18 ok happened again. After about 6 hours and 20 minutes. What looks also quite strange is, that the code should save values even when they are not updated(due to the error).
2022-03-24 19:33:26.980434,0.613562,
2022-03-24 19:33:47.029422,0.613759,
2022-03-24 20:59:49.383536,0.613759,
2022-03-24 21:00:09.455432,0.613759,
2022-03-24 21:00:29.516644,0.613759,
2022-03-24 21:00:49.596934,0.613759,
2022-03-24 21:01:09.678921,0.613759,
2022-03-24 21:01:29.766926,0.613759,
2022-03-24 21:01:49.852033,0.613759,
2022-03-24 21:02:09.930524,0.613759,
2022-03-24 21:02:30.010645,0.613759,
2022-03-24 21:02:50.085574,0.613759,
2022-03-24 21:03:10.137753,0.613759,
2022-03-24 21:03:30.186664,0.613759,
2022-03-24 21:03:50.245657,0.613759,
But as you can see in the data there is a gap. after that the error happen again. Could be like you said and the fault lies in my code but I'm not familliar enough with asyncio to know how to debug this. Any idea?
@Kiessar So you got again a « socket.send() raised exception. » ? Do you have the stack trace ? That way I can know if it failed when trying to reconnect.
@Atem18 I have clue but no prrof yet. I switched the log level to DEBUG (not that easy in the current state). What I think is happening:
The log level showed that before the socket.send() raised exception. is raised the following exception is thrown:
2022-03-25 14:20:29,793 kraky.ws DEBUG WebSockets connection closed error, let's reconnect anyway...
Which refers to: https://github.com/Atem18/kraky/blob/e9b9c9b947d0279eb227a7e238c6930593779776/kraky/ws.py#L73
But why is the errorhappening? My guess:
The message rate limit of a single WebSocket API connection will vary depending upon the load on the system. WebSocket API clients will receive the error message {"Error": "Exceeded msg rate"} when the message rate limit is exceeded.
Source
I'm using 20 subscriptions in the example and think that I'm running into the message rate limit. Unfortunately i couldn't find a way to show the error message. After that it could be that I run into the connection ban:
- Cloudflare imposes a connection/re-connection rate limit (per IP address) of approximately 150 attempts per rolling 10 minutes. If this is exceeded, the IP is banned for 10 minutes.
- Recommended reconnection behaviour is to (1) attempt reconnection instantly up to a handful of times if the websocket is dropped randomly during normal operation but (2) after maintenance or extended downtime, attempt to reconnect no more quickly than once every 5 seconds. There is no advantage to reconnecting more rapidly after maintenance during cancel_only mode.
I'm not sure what happens than in this part: https://github.com/Atem18/kraky/blob/e9b9c9b947d0279eb227a7e238c6930593779776/kraky/ws.py#L50-L60
What I read about was that if the error socket.send() raised exception.
occurs it is because send is called on a socket that is not open anymore. But not sure what happens. I will try to split the subscriptions onto multiple connections, but this will only mitigate the issue.
I fixed some issues in the code, if you need it for testing let me know. Should trigger quite quickly if the number of pairs is increased.
@Kiessar The main code is the function connect. As you can see, I am doing in infite loop and try catch in case an error occurs on the socket. In the try, I am checking if the socket is open, if it's not, I am reconnecting. If it's open, I get the message. If there is an "errorMessage" in the message, I log it in error and if there is no error, I pass it to the handler function. If you would get the error of the message, it would be logger as an error. In this issue https://github.com/aaugustin/websockets/issues/84, I can see that the person tried to add await asyncio.sleep(0) after websocket.send(...). I will see if I can reproduce the issue. And yes if you have code I can try, it's always nice. :)
@Atem18
I splitted it up and use a connection per pair now. The errorchanged now but first let me have you the code:
https://pastebin.com/EYD9JFVF
Starting with this parameters:
nohup python WSDataFetcher.py -p XBT/EUR,XBT/USD,PAXG/ETH,SOL/EUR -s 20 &
The new errors start with:
2022-03-26 04:00:12,974 kraky.ws DEBUG WebSockets connection closed error, let's reconnect anyway...
2022-03-26 04:00:12,974 kraky.ws DEBUG WebSockets connection closed error, let's reconnect anyway...
2022-03-26 04:00:12,976 kraky.ws DEBUG WebSockets connection closed error, let's reconnect anyway...
2022-03-26 04:00:12,978 kraky.ws DEBUG WebSockets connection closed error, let's reconnect anyway...
2022-03-26 04:00:13,867 kraky.ws DEBUG WebSockets connection closed error, message not sent...
2022-03-26 04:00:13,867 kraky.ws DEBUG WebSockets connection closed error, message not sent...
2022-03-26 04:00:13,867 kraky.ws DEBUG WebSockets connection closed error, message not sent...
goes on for a long while and finishs with:
2022-03-26 04:58:09,987 kraky.ws DEBUG WebSockets connection closed error, message not sent...
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/selector_events.py", line 848, in _read_ready__data_received
data = self._sock.recv(self.max_size)
MemoryError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/base_events.py", line 1733, in call_exception_handler
self.default_exception_handler(context)
MemoryError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/__init__.py", line 1085, in emit
msg = self.format(record)
File "/usr/lib/python3.8/logging/__init__.py", line 929, in format
return fmt.format(record)
File "/usr/lib/python3.8/logging/__init__.py", line 676, in format
record.exc_text = self.formatException(record.exc_info)
File "/usr/lib/python3.8/logging/__init__.py", line 626, in formatException
traceback.print_exception(ei[0], ei[1], tb, None, sio)
File "/usr/lib/python3.8/traceback.py", line 103, in print_exception
for line in TracebackException(
MemoryError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "/usr/lib/python3.8/asyncio/selector_events.py", line 805, in _read_ready
self._read_ready_cb()
File "/usr/lib/python3.8/asyncio/selector_events.py", line 854, in _read_ready__data_received
self._fatal_error(exc, 'Fatal read error on socket transport')
File "/usr/lib/python3.8/asyncio/selector_events.py", line 705, in _fatal_error
self._loop.call_exception_handler({
File "/usr/lib/python3.8/asyncio/base_events.py", line 1740, in call_exception_handler
logger.error('Exception in default exception handler',
File "/usr/lib/python3.8/logging/__init__.py", line 1475, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3.8/logging/__init__.py", line 1589, in _log
self.handle(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1599, in handle
self.callHandlers(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1669, in callHandlers
lastResort.handle(record)
File "/usr/lib/python3.8/logging/__init__.py", line 954, in handle
self.emit(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1093, in emit
self.handleError(record)
File "/usr/lib/python3.8/logging/__init__.py", line 1007, in handleError
traceback.print_exception(t, v, tb, None, sys.stderr)
File "/usr/lib/python3.8/traceback.py", line 103, in print_exception
for line in TracebackException(
MemoryError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "WSDataFetcher.py", line 216, in <module>
loop.run_forever()
File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
self._run_once()
File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
handle._run()
File "/usr/lib/python3.8/asyncio/events.py", line 85, in _run
cb = format_helpers._format_callback_source(
File "/usr/lib/python3.8/asyncio/format_helpers.py", line 24, in _format_callback_source
source = _get_function_source(func)
MemoryError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "WSDataFetcher.py", line 218, in <module>
traceback.print_exc()
File "/usr/lib/python3.8/traceback.py", line 163, in print_exc
print_exception(*sys.exc_info(), limit=limit, file=file, chain=chain)
File "/usr/lib/python3.8/traceback.py", line 103, in print_exception
for line in TracebackException(
MemoryError
Next I will check my code for issues if the memoryError is caused in my domain.
Just a question, is it necessary to close the websockets graefully after websockets.open
is false?
Just a question, is it necessary to close the websockets graefully after websockets.open is false?
Just like any connection, it's better if you can. There is a method that you can call which is disconnect, to close the websocket if you want. If not, you can just kill your program and the connection will be closed anyway. Also I pushed a new version where you can configure the logging level.
@Atem18 Just two things I came across while thinking what to do next: I think there is something that cloeses the conncetion form server side (e.g. message limit). My guess why I'm not able to see the error message is, that the connection is not open anymore, so websocket.open returns false but there are still messages in the pipe that could be received. By replacing the websocket, the pipe is droped.
Second thing is, wouldn't it be a good idea to add a sleep in each exception? This could prevent an IP ban, due to too many reconnections.
@Kiessar Indeed, that could be an explanation but that would mean that the server is closing the connection before sending you the error. And yes, it would be a good idea, I will add it and add the possibility to change it.
@Atem18 Yes but think of the server is sending you like 50 messages and afterwards closing the connection. The connection state is checked after each recv (my assumption is one recv for each message). At some point the connection is already closed while it still tries to process all messages. Is it actually neccessary to check the connection? recv should raise an Exception/Error when pipe is drained and connection is closed.
@Kiessar I pushed a new release with support of timeout and sleep_time. Please let me know if it can help.
@Atem18 Just a small report. First time it ran about 24h after that the socket.send() was raised again. Unfortunately I had forgot to reset the log level (updating overwrote my changes). I corrected that and now it runs more than 24h. I think if it crashes one more time, I will increase the timeout to 10 seconds. If than still happens anything, I will let you know.
@Kiessar No problem. :)
@Kiessar Any news ?
@Kiessar Any news ?
@Atem18 It crashed again, but some time ago and I hadn't had the time to check on it. I will try to let it run again next day's. Thank you for your interest in this.
@Kiessar Ok good, please let me know your findings.
I pushed a new version with an updated version of websockets. Maybe it will help.
I pushed a new version with an updated version of websockets. Maybe it will help.
Ok I started in the morning but crashed caused by an error of my code. Will give it another try soon and let you know the result. Thanks again
still running...
@Atem18 Unfortunately error still persists. Set all the timeouts to 10 and still gettingthe error: socket.send() raised exception.
@Kiessar Ok thanks, I will try to reproduce it as well and see if something can be done at the client's side.
@Kiessar I tried to run a simple implementation and my script got killed by the OOM killer.
Did you also got an increase of memory over time ?
@Atem18 Yes, one time I got an memory error, but I thought it was caused by my own implementation.
Good to know but in that case, it's just simply listening the websocket and printing the output.
@Atem18 Because I'm starting to need the functionality more and more I will start to dive in more and try to solve the issue. My latest debug log is the following:
2022-06-15 08:14:59,008 kraky.ws DEBUG WebSockets connection closed error - retrying connection in 5 sec.
2022-06-15 08:14:59,028 kraky.ws DEBUG WebSockets connection closed error - retrying connection in 5 sec.
2022-06-15 08:14:59,110 kraky.ws DEBUG WebSockets connection closed error - retrying connection in 5 sec.
2022-06-15 08:14:59,137 kraky.ws DEBUG WebSockets connection closed error - retrying connection in 5 sec.
2022-06-15 08:14:59,298 kraky.ws DEBUG WebSockets connection closed error - retrying connection in 5 sec.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
The memory error probably does occur because it prints endlessly multiple times per second the line "socket.send() raised exception.". The memory error is therefore probably not the cause but just another symptom.
@Kiessar For the memory error, yes it's probably that.
If I understand well, you are opening 5 websockets connections, right ?
And 4 of them did not reconnected ?
@Atem18 Sorry didn't make that clearly enough. It goes repeating the line
"socket.send() raised exception."
for about 500mB (so thousands of lines repeated). So the number of lines was just arbitrary cut off and are more than those 4 (so i guess that all of them don't reconnect)
@Kiessar Ok I see what you mean.
So maybe it's linked to https://github.com/aaugustin/websockets/issues/84 .
I will try to add await asyncio.sleep(0) after websocket.send(...) and see if it will help.
@Kiessar A new release is up with the sleep after the send. Can you please test it and tell me ?
Thanks a lot in advance.
@Atem18 New error and it crashed my server due to overload and diskspace consumption:
2022-06-20 01:54:53,144 kraky.ws DEBUG WebSockets connection closed error - retrying connection in 20 sec.
2022-06-20 01:56:13,824 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-06-20 01:56:13,824 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-06-20 01:56:13,824 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-06-20 01:56:13,825 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-06-20 01:56:13,825 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-06-20 01:56:13,825 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-06-20 01:56:13,825 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-06-20 01:56:13,825 kraky.ws DEBUG WebSockets connection closed error - message not sent.
[...]
Diskspace was used because the error was written in the log with an insane rate.
@Kiessar It seems another error this time. Unless I am wrong, it went into an infinite loop.
Let me check again my logic and see if I can find the flaw.
@Kiessar quick question, can I have a snippet of your code that raise the issue ?
Thanks a lot in advance.
@Atem18
Script:
https://pastebin.com/scZ0qA27
Command:
nohup python WSDataFetcher.py -p RARI/EUR,ADA/EUR,PAXG/ETH,SOL/EUR,RARI/XBT,XBT/EUR,ADA/USD -s 10 &
My best guess is still, that it runs into rate limits. Therefore it highly depends on the current market usage. If I'm correct adding more pairs would lead to quicke errors.
@Kiessar Thanks I will have a look.
Also there is still the issue that at some point, the memory usage is increasing but now that I have proper monitoring I can see that it’s not increasing over time but in less than 5 minutes !
So maybe there is a bug in my library, maybe there is one in the websockets library but maybe Kraken’s Websocket API have some issues that the libraries are not enjoying.
So let’s see if we can find anything useful…
@Atem18 Are you sure thats not my code whats causing the memory consumption? Just don't want to waste your time. If you like I could strip down everything regarding storing the data, so that only the connection is used to get data.
@Kiessar Yes, I also have the issue with the example in my README.md
But yes, if you can test/monitor on your side, it would be great !
Hi @Kiessar, do you still have the issue ?
Thanks a lot in advance.
Hi @Atem18 , I will check in coming days and update as soon as I have news.
@Kiessar Thanks a lot
Hi @Atem18, sorry for the late reply: It ran for about 4 days and than into the same situation:
2022-09-16 19:28:38,741 kraky.ws DEBUG WebSockets connection closed ok - retrying connection in 20 sec.
2022-09-16 19:29:59,465 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-09-16 19:29:59,465 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-09-16 19:29:59,466 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-09-16 19:29:59,466 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-09-16 19:29:59,466 kraky.ws DEBUG WebSockets connection closed error - message not sent.
2022-09-16 19:29:59,466 kraky.ws DEBUG WebSockets connection closed error - message not sent.
[...]
Still thinking, it is some rate limit issue that is building up
@Kiessar Thanks for the feedback. Yes, there could be some rate limit from Kraken.
I propose to close this issue as there is no longer the error "socket.send() raised exception".
Hi there, thank you for the cool tool. I'm trying to gather data with kraky and getting the erro:
socket.send() raised exception.
Here the base of my code to make sure I didn't mess up anything else.