Closed Flowelfox closed 3 years ago
My anwser is not cool: I know, there is a bug inside, but I cant trace it till now. I dont know what and why it happens, but in my tests i often end up with active streams which are eceiving nothing. If i set a restart request the original thread does not get closed, but a new extra one is created. so there are zombi threads after a while and the lib cant handle it atm - i think i would need a couple of days for testing and diving into the code, but i am too busy atm to get that time.
code = 1006 is nothing i can really control. I think, its something happening on the network or there are to less local resources (cpu/ram) and also sometimes binance seems to be very unstable. the same code runs 2 weeks with zero restart and then i have days with a lot of restarts per day.
I want to crate something that can proof if a thread and its stream is really working, if not, kill and restart.
Maybe logging needs to get a clever update too, because atm the logs dont help me to find the reason for dead streams.
how much cpu and ram does you plattform have? how much % of the system are you using in average?
My local machine has 32GB or ram and 6 core, 12 threads CPU. I am using only 10-20% of my cpu.
On VPS there is only 1 core, 1 thread and 2GB of ram. Full application + web site using 50% of CPU in avarege and ~600MB of memory.
Did you read the link i posted about 1006 in the other thread?
1006 is something thats happening outside of the lib. the lib is getting this signal from the OS, the reason is that the connection is not established anymore... for any reason. This can happen through physical line probs, problems on binance servers, problems with your local resources, what ever...
Its really hard for me to find the specific reason and I think we have to deal with that. If you have a better idea lets talk about it.
First I loved the idea of one mega multiplex connection all pairs and all stream types, but then i recognized that such channels are very unstable. the less channel types I mixed, the more stable they were. I dont know the reason.
Also if I had to less CPU on my server I had a lot of side effects...
Please take a look at https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/example_stream_everything.py - here i create all kind of kline streams with all available pairs - in separate streams. the most stable way i found out. sometimes 14 without a single restart on my VPS.
What does happen if you run that script on your server where you have the problems?
This script is running on my demo server: https://ubwa-demo.lucit.co/print_summary.php
Can we close this?
Yes, but issue still exist
Did you read the link i posted about 1006 in the other thread?
1006 is something thats happening outside of the lib. the lib is getting this signal from the OS, the reason is that the connection is not established anymore... for any reason. This can happen through physical line probs, problems on binance servers, problems with your local resources, what ever...
Its really hard for me to find the specific reason and I think we have to deal with that. If you have a better idea lets talk about it.
First I loved the idea of one mega multiplex connection all pairs and all stream types, but then i recognized that such channels are very unstable. the less channel types I mixed, the more stable they were. I dont know the reason.
Also if I had to less CPU on my server I had a lot of side effects...
Please take a look at https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/example_stream_everything.py - here i create all kind of kline streams with all available pairs - in separate streams. the most stable way i found out. sometimes 14 without a single restart on my VPS.
What does happen if you run that script on your server where you have the problems?
What are your thoughts about that?
Please try an other setup: Only one channel like kline_1m and then all symbolpairs (but less than 1024 per stream)
create_stream('kline_1m', ['btcusdt', 'ethusdt', 'bchusdt', 'xrpusdt', 'eosusdt', 'dashusdt', 'xmrusdt', 'zecusdt', 'xtzusdt', 'adausdt'])
create_stream('kline_3m', ['btcusdt', 'ethusdt', 'bchusdt', 'xrpusdt', 'eosusdt', 'dashusdt', 'xmrusdt', 'zecusdt', 'xtzusdt', 'adausdt'])
create_stream('kline_5m', ['btcusdt', 'ethusdt', 'bchusdt', 'xrpusdt', 'eosusdt', 'dashusdt', 'xmrusdt', 'zecusdt', 'xtzusdt', 'adausdt'])
create_stream('kline_1h', ['btcusdt', 'ethusdt', 'bchusdt', 'xrpusdt', 'eosusdt', 'dashusdt', 'xmrusdt', 'zecusdt', 'xtzusdt', 'adausdt'])
create_stream('kline_1d', ['btcusdt', 'ethusdt', 'bchusdt', 'xrpusdt', 'eosusdt', 'dashusdt', 'xmrusdt', 'zecusdt', 'xtzusdt', 'adausdt'])
I tested with a few different VPS and settings.
1006 after too many kline streams... in my tests the problem is always the CPU power. when i scale it up, then the streams are getting stable....
Check this or we will delete your issue. (fill in the checkbox with an X like so: [x])
Select one:
Environment
Operating System? (include version)
Python Version Requirement
Exact Python Version?
Pip Version?
Dependencies
Run
pip list > pip_list.txt
and upload the file.UNICORN Binance WebSocket API Version?
Description of your issue
I am using 24 pairs and 12 kline stream for each pair. This all is starting in one BinanceWebSocketApiManager. After start i constantly start getting critical errors "code = 1006 (connection closed abnormally [internal]), no reason" ~every second.
If i start 3 BinanceWebSocketApiManager with pairs grouped as 10, 10 and 4 i am getting same error but much less often, one in like 5-10 minutes.
This started on 02.09 morning from version 1.16.4dev and continues now. Also i noticed that after crash only one or two specific streams starting, not all of them.
Logs without groups, 24 pairs, 1 manager:
Logs with grouped symbols, 6 managers, 5 per manager: