Closed jon4hz closed 2 years ago
Hi! What happens in the logs before this problem pops up? Best regards, Oliver
This problem is interesting, do you know a way how to reproduce it?
How may open files do you after, if this happens on your system? lsof | wc -l
What is the root cause?
Its trying to restart, why does the restart not work?
Hi and thanks for the fast reply. I'm going to send the rest of the log file in a few hours. I'm running the websocket in a container to sync the order book. I detected the issue because the order book suddenly stopped changing for more that 40 seconds and that's very unlikly. Sadly I don't know how to reproduce this problem but the websocket for other containers stopped working a few minutes later as well.
Hi, I just saw that the whole server had a very high cpu usage before this problem popped up. Also I found the line in the log file that initially caused the problem:
2020-11-17 22:54:00,049 [CRITICAL] 1 140390971710240 unicorn_binance_websocket_api_socket: BinanceWebSocketApiSocket.start_socket(d8aad01d-2012-4872-9d4c-a9306dbb88ce, ['depth'], ['xrpusdt']) - Exception ConnectionClosed - error_msg: code = 1006 (connection closed abnormally [internal]), no reason
As you may notice it's not from the same container but as I already said, I have multiple containers and all had the same issue. This one just disconnected a few hours after the bch-container.
could you provide me one full logfile? its possible to upload the text file here in the comment by drag & drop.
Here you go. Thanks again for your fast answers. Sadly I had to compress the file, it was to large for the upload.
Hm, I think a problem is that the app doenst get enough cpu power! I will study the log if i have more time to make a better handling for it, but I guess the only good solution is to power up your system!
Hello!
I think its very clear, from 2020-11-17 01:34:08 to 2020-11-17 22:53:59 your CPU is under full load.
Then you get the 1006 error, which means the connection has gone for unknown reasons.
Then it the thread that did not die detect that the stream isnt running anymore and tries to restart it, but it looks like the python interpreter isnt running anymore "cannot schedule new futures after interpreter shutdown" so schedule a new futures doesnt work and the finally
code part keeps up this loop.
As mentioned before, please power up the system, because if you have a constant backlog on the stream your are loosing data more often than necessary.
The only handling i can add is a graceful shutdown and a better log msg.
Best regards!
Hi, thanks for the fast fix. I'm planing to switch to a better server and when I do so I will also upgrade unicorn to the newest version.
Your welcome! :+1:
Hi, by now I think that the issue, that the websocket fails to reconnect after an error 1006 followed by an interpreter shutdown is related to python 3.9. In your telegram group was a conversation about the same issue and it could be fixed by downgrading to python 3.8. My scripts also didn't fail since I downgraded to python 3.8. Probably you should mention in your docs somewhere that python 3.9 is currently unstable and better not to be used.
I leave it open and wait for further confirmations that its a python 3.9 issue. thanks for sharing the info!
can confirm. python3.9 reproduces this error for me.
can confirm. python3.9 reproduces this error for me.
I can confirm this as well.
I have some findings on this topic. On my bot project I got the error at the beginning of the execution and my script couldn't run at all. This is a minimal way for reproducing the issue:
from unicorn_binance_websocket_api.unicorn_binance_websocket_api_manager import BinanceWebSocketApiManager
API_KEY = '*'
API_SECRET = '*'
binance_websocket_api_manager = BinanceWebSocketApiManager(
exchange='binance.com-futures',
)
binance_websocket_api_manager.create_stream(
channels=['!userData'],
markets=['arr'],
api_key=API_KEY,
api_secret=API_SECRET
)
$ python3 sample.py
CRITICAL:root:BinanceWebSocketApiManager._create_stream_thread() stream_id=db732ee8-01bc-492f-98e4-caa0e43654ea - RuntimeError error_msg: - cannot schedule new futures after interpreter shutdown - stopping and shutting down - read https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/issues/131 for further information!
What I found out is that a simple while
loop at the end of the script is enough to workaround the problem:
from unicorn_binance_websocket_api.unicorn_binance_websocket_api_manager import BinanceWebSocketApiManager
import time
API_KEY = '*'
API_SECRET = '*'
binance_websocket_api_manager = BinanceWebSocketApiManager(
exchange='binance.com-futures',
)
binance_websocket_api_manager.create_stream(
channels=['!userData'],
markets=['arr'],
api_key=API_KEY,
api_secret=API_SECRET
)
while True:
time.sleep(1)
I can't be sure why it works but the issue was fixed on my project once I added a while
loop at the end of the script for the binance_websocket_api_manager.print_summary()
method so I could consume the websocket data normally. Maybe the loop prevents the interpreter from shutting down at the beginning of the execution so the python process keeps running.
Another thing I found out is that the issue seems to be related with the use of threading.Thread()
method. If I replace threading.Thread()
with a while
loop to consume the websocket everything works fine. However, even using threading.Thread()
you can workaround the issue if you use an empty while
loop at the end.
Could be solved through https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/pull/176
Some feedback would be nice!
Could be solved through #176
Some feedback would be nice!
Still not solved. Try this script with Python 3.9:
from unicorn_binance_websocket_api.manager import BinanceWebSocketApiManager
API_KEY = '*'
API_SECRET = '*'
binance_websocket_api_manager = BinanceWebSocketApiManager(
exchange='binance.com-futures',
)
binance_websocket_api_manager.create_stream(
channels=['!userData'],
markets=['arr'],
api_key=API_KEY,
api_secret=API_SECRET
)
How did you test? Did you also update requirements?
python3 -m pip install -r requirements.txt --upgrade
Then you should be using websockets 9.1 instead of 8.1 which could make the difference....
Atm i dont have python3.9 installed anywhere....
How did you test? Did you also update requirements?
python3 -m pip install -r requirements.txt --upgrade
Then you should be using websockets 9.1 instead of 8.1 which could make the difference....
I'm using v9.1. I've updated websockets via pipenv.
Thanks!
websockets 10.0 released, it drops support for python 3.6 and adds support for python 3.10.
its worth to take a look if that bug is still there.
can anyone update to websockets 10.0 and test it?
https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/pull/195
updated to websockets 10: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/releases/tag/1.34.0
feedback is appreciated
updated to websockets 10: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/releases/tag/1.34.0
feedback is appreciated
Issue still present with websockets 10 and Python 3.9.8.
websockets 10.1 is released with this commit: https://github.com/aaugustin/websockets/commit/448e777adc7d8cf0cbc0bd585959bbdfad682926
not sure if it is related to our shutdown problem but its worth a try!
websockets 10.1 is released with this commit: aaugustin/websockets@448e777
not sure if it is related to our shutdown problem but its worth a try!
Still not working with websockets 10.1. I just found this StackOverflow question that seems to be related to the issue.
Hello,
I'm seeing this error intermittently. I often get the following message, but then it works sometimes:
CRITICAL:root:BinanceWebSocketApiManager._create_stream_thread() stream_id=e4397b60-4d09-4f3a-9c3a-94dbb3b2970f - RuntimeError error_msg: - cannot schedule new futures after interpreter shutdown - stopping and shutting down - read https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/issues/131 for further information!
python --version Python 3.9.9
pip list Package Version
aniso8601 9.0.1 attrs 21.2.0 certifi 2021.10.8 cffi 1.15.0 charset-normalizer 2.0.9 cheroot 8.5.2 click 8.0.3 colorama 0.4.4 cryptography 36.0.1 dateparser 1.1.0 Flask 2.0.2 Flask-RESTful 0.3.9 idna 3.3 itsdangerous 2.0.1 jaraco.functools 3.5.0 Jinja2 3.0.3 MarkupSafe 2.0.1 more-itertools 8.12.0 numpy 1.21.5 pandas 1.3.5 pathlib 1.0.1 pip 21.3.1 psutil 5.8.0 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycparser 2.21 pyOpenSSL 21.0.0 python-dateutil 2.8.2 pytz 2021.3 pytz-deprecation-shim 0.1.0.post0 regex 2021.11.10 requests 2.26.0 service-identity 21.1.0 setuptools 58.1.0 six 1.16.0 termcolor 1.1.0 tzdata 2021.5 tzlocal 4.1 ujson 5.1.0 unicorn-binance-rest-api 1.3.0 unicorn-binance-websocket-api 1.34.2 unicorn-fy 0.11.0 urllib3 1.26.7 websocket-client 1.2.3 websockets 10.0 Werkzeug 2.0.2
I've also tried Python 3.10 however there unicorn won't install because of an issue with pathlib. It would go through each version, and fail each time with something like Discarding... files.pythonhosted.org.....pathlib-1.0.1.tar.gz#sha256=...
Thanks,
John
I have the same issue with python 3.10. It is also intermittent.
The pathlib problem with python 3.10 is fixed since 1.35.0!
I didn't follow the entire thread... Does this mean, we can close the issue?
Short: No! :/
Long: 45by90 was not able to test with python 3.10 because in 3.10 is pathlib a standard lib and does not need to get installed as a dependencie with setup.py/requirements.txt anymore. So installation in 3.10 has thrown errors till 1.35.0 and papapon was able to do the installation and he said the issue is still there.
@oliver-zehentleitner did you take a look at the link in my previous comment?
I don't know why, I forgot.
I test that, thank you @xilopaint
2nd line of the sample code broke down. I am using MacMini, Python3.9, Websockets 10.1 and every dependency seems properly installed.
BinanceWebSocketApiManager._create_stream_thread() stream_id=096f3190c49e-c11a-8aee-8cd2-5dc9f8f5 - RuntimeError error_msg: - cannot schedule new futures after interpreter shutdown - stopping and shutting down
Ok, let's put this together:
BinanceWebSocketApiManager._create_stream_thread() stream_id=096f3190c49e-c11a-8aee-8cd2-5dc9f8f5 - RuntimeError error_msg: - cannot schedule new futures after interpreter shutdown - stopping and shutting down
BinanceWebSocketApiManager._create_stream_thread() stream_id=ca55a8798f3f-ee62-bc8e-0fae-97eebdbb error: 7 - can't register atexit after shutdown
A possible solution is adding threading.join()
as described here to this object: https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/5d749831c6f84554fa548f79535990c0d78b3c2d/unicorn_binance_websocket_api/manager.py#L1208
I am thinking about where to place this join()
, actually we dont need it because we try to stop threads from inside the thread. Do we know a way to replicate this error? Does it only need a running main thread?
I made a test script as described by @xilopaint but with a public stream instead of userData... its easier to share and work with because we dont need a key and secret.
from unicorn_binance_websocket_api.manager import BinanceWebSocketApiManager
import logging
import time
logging.getLogger("unicorn_binance_websocket_api")
logging.basicConfig(level=logging.DEBUG,
format="{asctime} [{levelname:8}] {process} {thread} {module}: {message}",
style="{")
ubwa = BinanceWebSocketApiManager(exchange="binance.com-futures")
ubwa.create_stream("kline_1m", "btcusdt", stream_label="kline_1m")
It is throwing the error message 2 of the above list. Which seems to be some kind of race condition:
ubwa.create_stream()
wants to start a thread, but this just gets triggered and the main thread exists because there is no more code to execute. In the meantime the python interpreter is not able to fully start the new thread and throws a runtime error.
Simply adding time.sleep(0.01)
is enough on my system to get the thread started and avoiding the error. So we can show a better error or we can add a sleep to create_stream before the return
command of the function.
here is the test file: https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/blob/master/dev_test_issue_131.py
2nd line of the sample code broke down. I am using MacMini, Python3.9, Websockets 10.1 and every dependency seems properly installed.
BinanceWebSocketApiManager._create_stream_thread() stream_id=096f3190c49e-c11a-8aee-8cd2-5dc9f8f5 - RuntimeError error_msg: - cannot schedule new futures after interpreter shutdown - stopping and shutting down
Can you exactly describe me how i can reproduce this error?
https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/releases/tag/1.36.1 includes a fix for error 2 of the list above!
2nd line of the sample code broke down. I am using MacMini, Python3.9, Websockets 10.1 and every dependency seems properly installed. BinanceWebSocketApiManager._create_stream_thread() stream_id=096f3190c49e-c11a-8aee-8cd2-5dc9f8f5 - RuntimeError error_msg: - cannot schedule new futures after interpreter shutdown - stopping and shutting down
Can you exactly describe me how i can reproduce this error?
You can reproduce this error by using the code I suggested in this previous comment.
2nd line of the sample code broke down. I am using MacMini, Python3.9, Websockets 10.1 and every dependency seems properly installed. BinanceWebSocketApiManager._create_stream_thread() stream_id=096f3190c49e-c11a-8aee-8cd2-5dc9f8f5 - RuntimeError error_msg: - cannot schedule new futures after interpreter shutdown - stopping and shutting down
Can you exactly describe me how i can reproduce this error?
You can reproduce this error by using the code I suggested in this previous comment.
I allways get this error from this test: "can't register atexit after shutdown".
@xilopaint can we talk on telegram please?
@xilopaint can we talk on telegram please?
I've sent you a message.
the same script produces on mac and linux different errors. on linux its fixed with release 1.36.1.
I need some time on my macbook to investigate this ...
the same script produces on mac and windows different errors. on linux its fixed with release 1.36.1.
I need some time on my macbook to investigate this ...
Have you checked the workaround suggested by me in this comment?
Yes, but only on linux. there i was able to fix it. the reason, the main thread has to life long enough, till the thread is fully started, then it can close. so we just wait a little bit before proceeding and the error on linux is gone. but not on mac and i was not on a macos till now...
Yes, but only on linux. there i was able to fix it. the reason, the main thread has to life long enough, till the thread is fully started, then it can close. so we just wait a little bit before proceeding and the error on linux is gone. but not on mac and i was not on a macos till now...
Ok. Just to let you know that workaround works for me on macOS.
you mean its solved?
you mean its solved?
Dude, I posted that workaround 11 months ago. Have you never tested it before? Yes, it solves the issue for me, but I didn't try to implement it on ubwa code and I don't know if it's the best solution for the issue.
OK :)
I tested your code on linux and there the workaround helped me to find the solution. the thread needs to be completlly started before exiting the main thread, with while True
you reach that. on linux my patch fixed it... i thought you meant in the post before it works on mac too...
the while true is not possible in the lib, because create_stream() would become a blocking function then... i will investigate this.
Check this or we will delete your issue. (fill in the checkbox with an X like so: [x])
Select one:
Environment
What kind of internet connection do you have?
~1-2 GB/s
Average system load (CPU)
45%
Hardware specification
VPS, Vultr
Operating System? (include version)
Options
Python Version Requirement
Exact Python Version?
Pip Version?
Dependencies
Run
pip list > pip_list.txt
and upload the file.UNICORN Binance WebSocket API Version?
Description of your issue