oliver-zehentleitner
commented
3 years ago Hi am11y!
The point is, that each stream is running within its own thread. Even if that threads are working on different CPU cores, unfortunately Pythons GIL only executes one thread after the next in a cycle, not really parallel.
For example if you have 3 CPU cores you have 3 x 100% CPU power. But all threads of one python application can finally consume only 1 x 100% CPU. Even if the other 200% are free, the Python app cant use it!
It might depend of the subscriptions you have, but in my tests in peak times the transmitted data from binance is x 3 of the average traffik:
So if you have in average 100% CPU usage the received items cant get processed as fast as they come in - the backlog is growing and if the backlog gets too big, the streams disconnects with error 1006. Sure it reconnects afterward, but you are also losing data more often than it would be necessary.
This is not easy to detect for most people, like you said, your system is not under full load, but your script can not acquire more!
In the future I want to replace threads with independent processes, they dont have the GIL restrictions.
This logging is new and I will try to find the right balance to avoid spamming the logs. In the next release it will be much less log entries through https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/issues/124
If you have a lot of this entries, i bet you have also a lot of avoidable reconnects...
am11y
I get it constantly on many different systems, and none of those are very old.
Plus nowhere on those systems do I really see high CPU usage reported from the system.
It just spams my standard output and really gets on my nerves. I have to delete it manually with each update.
Could you perhaps consider removing it?
Thanks.