search

LUCIT-Systems-and-Development / unicorn-binance-websocket-api

A Python SDK by LUCIT to use the Binance Websocket API`s (com+testnet, com-margin+testnet, com-isolated_margin+testnet, com-futures+testnet, com-coin_futures, us, tr, dex/chain+testnet) in a simple, fast, flexible, robust and fully-featured way.
https://unicorn-binance-websocket-api.docs.lucit.tech/
Other
683 stars 164 forks source link

module 'asyncio.base_futures' has no attribute 'InvalidStateError' #83

Closed Flowelfox closed 3 years ago

Flowelfox commented 4 years ago

Check this or we will delete your issue. (fill in the checkbox with an X like so: [x])

Select one:

Environment

Operating System? (include version)

Python Version Requirement

Exact Python Version?

Python 3.8.3

Pip Version?

pip 20.1.1

Dependencies

Package                       Version    Location
----------------------------- ---------- ---------
alembic                       1.4.2
aniso8601                     8.0.0
attrs                         19.3.0
autobahn                      20.4.3
Automat                       20.2.0
certifi                       2020.4.5.1
cffi                          1.14.0
chardet                       3.0.4
cheroot                       8.3.0
click                         7.1.2
colorama                      0.4.3
constantly                    15.1.0
cryptography                  2.9.2
dateparser                    0.7.4
Flask                         1.1.2
Flask-RESTful                 0.3.8
flask-restx                   0.2.0
Flask-SQLAlchemy              2.4.3
hyperlink                     19.0.0
idna                          2.9
incremental                   17.5.0
inprofitapi                   0.1        /inprofit
itsdangerous                  1.1.0
jaraco.functools              3.0.1
Jinja2                        2.11.2
jsonschema                    3.2.0
Mako                          1.1.3
MarkupSafe                    1.1.1
more-itertools                8.3.0
pathlib                       1.0.1
pip                           20.1.1
psutil                        5.7.0
psycopg2-binary               2.8.5
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycparser                     2.20
PyHamcrest                    2.0.2
pyOpenSSL                     19.1.0
pyrsistent                    0.16.0
python-binance                0.7.5
python-dateutil               2.8.1
python-editor                 1.0.4
pytz                          2020.1
regex                         2020.5.14
requests                      2.23.0
service-identity              18.1.0
setuptools                    47.1.1
six                           1.15.0
SQLAlchemy                    1.3.17
Twisted                       20.3.0
txaio                         20.4.1
tzlocal                       2.1
ujson                         3.0.0
unicorn-binance-websocket-api 1.15.0
unicorn-fy                    0.4.1
urllib3                       1.25.9
websocket-client              0.57.0
websockets                    8.1
Werkzeug                      1.0.1
wheel                         0.34.2
zope.interface                5.1.0

UNICORN Binance WebSocket API Version?

Did you upgrade to the latest release version with `pip install unicorn-binance-websocket-api --upgrade`?
No. I am using version 1.15.0

Description of your issue

Debug log when streams crashed:

2020-06-08 16:05:14,701 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 16:35:14,721 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
2020-06-08 16:35:15,461 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 16:53:47,277 - root - DEBUG - BinanceWebSocketApiSocket->start_socket(76622917-8447-4e2a-b723-c2df11feebc2, ['kline_1m', 'kline_3m', 'kline_5m', 'kline_15m', 'kline_30m', 'kline_1h', 'kline_2h', 'kline_4h', 'kline_6h', 'kline_8h', 'kline_12h', 'kline_1d'], ['btcusdt', 'ethusdt', 'bchusdt', 'xrpusdt', 'eosusdt', 'ltcusdt', 'trxusdt', 'etcusdt', 'linkusdt', 'xlmusdt', 'adausdt', 'xmrusdt', 'dashusdt', 'zecusdt', 'xtzusdt', 'bnbusdt', 'atomusdt', 'ontusdt', 'iotausdt', 'batusdt', 'vetusdt', 'neousdt', 'qtumusdt', 'iostusdt']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError'
2020-06-08 17:05:15,521 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
2020-06-08 17:05:15,928 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 17:35:15,973 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
2020-06-08 17:35:16,543 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 17:46:30,840 - root - DEBUG - BinanceWebSocketApiSocket->start_socket(27264e1f-532d-4293-933c-38e6f02cb0ad, ['arr'], ['!miniTicker']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError'
2020-06-08 17:46:31,427 - root - DEBUG - BinanceWebSocketApiSocket->start_socket(2ca0094e-8dea-46cf-af8d-8ac704876335, ['arr'], ['!userData']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError'
2020-06-08 18:05:16,621 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
2020-06-08 18:05:17,024 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 18:35:17,065 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
2020-06-08 18:35:17,539 - root - INFO - sent listen_key keepalive ping for stream_id=2ca0094e-8dea-46cf-af8d-8ac704876335
2020-06-08 19:05:17,635 - root - DEBUG - BinanceWebSocketApiRestclient->keepalive_listen_key(xImwgXC6A2tBWaxJR76frQLPJkwIEuttb7InMUaCgK8oTq8wU3PliQbEoq7PRysc)
Kline stream, and then other websocket streams (miniTicker, UserData) will shutdown without any error, exception after some time of work. It can happen in hours or days. Last time it worked almost 5 days.
I found some info about what happened in debug log, i added log above.
Last candle received in 16:53. After that no new data was added to database.
I assume that i must get exception from unicorn_binance_websocket_api_connection.py:268, but somehow i didn't have InvalidStateError.
Also it worries me because it was shutted down silently. No errors, no critical error or anything else. I was 

I ask for help to understand what caused this issue, my code or something wrong in library?
How i can avoid getting this error and stream crashing?
Flowelfox commented 4 years ago

I found that in Python 3.8.0 and above InvalidStateError does not exists:

(.venv3.8) flowelcat@Flowelcat-PC:~/PycharmProjects/inprofitapi$ python test.py 
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    from asyncio.futures import InvalidStateError
ImportError: cannot import name 'InvalidStateError' from 'asyncio.futures' (/usr/lib/python3.8/asyncio/futures.py)
(.venv3.8) flowelcat@Flowelcat-PC:~/PycharmProjects/inprofitapi$ python --version
Python 3.8.3
oliver-zehentleitner commented 4 years ago

Thanks for the good report! I added a restart handling for this error, please can you test it? pip install https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/tarball/master --upgrade Please proof that you are on 1.16.1.dev!

I am not shure if the code I added is really good, maybe we have to filter out some other attribute errors... lets test it :)

Best regards, Oliver

Flowelfox commented 4 years ago

Hi, updated library and restarted application recently.

Version proof:

inprofitapi | 2020-06-09 20:04:38,817 - root - INFO - Initializing flask application...
inprofitapi | 2020-06-09 20:04:38,891 - root - INFO - Setting up kline subscription...
inprofitapi | 2020-06-09 20:04:38,892 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1 started ...
inprofitapi | 2020-06-09 20:04:39,031 - root - INFO - Starting missing candles loader...
inprofitapi | 2020-06-09 20:04:39,036 - root - INFO - Setting up mark price subscription...
inprofitapi | 2020-06-09 20:04:39,038 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1 started ...
inprofitapi | 2020-06-09 20:04:39,183 - root - INFO - Setting up mini ticker subscription...
inprofitapi | 2020-06-09 20:04:39,184 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1 started ...
inprofitapi | 2020-06-09 20:04:39,284 - root - INFO - Setting up user data subscription...
inprofitapi | 2020-06-09 20:04:39,300 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1 started ...
inprofitapi | 2020-06-09 20:04:39,431 - root - INFO - Starting flask application...

And

flowelcat@vps729526:~/inprofit$ docker container exec -it inprofitapi python
Python 3.8.3 (default, Jun  3 2020, 19:49:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from unicorn_binance_websocket_api.unicorn_binance_websocket_api_manager import BinanceWebSocketApiManager
>>> 
>>> binance_websocket_api_manager = BinanceWebSocketApiManager()
>>> print(binance_websocket_api_manager.get_version())
1.16.1
>>>

A will monitor logs and candles from stream and notify here if something goes wrong.

oliver-zehentleitner commented 4 years ago

1.16.1 is WRONG! It must be 1.16.1.dev!

Try to deinstall all unicorn installations via pip and then run the installation of the dev stage again!

Flowelfox commented 4 years ago

Ok, changed version to 1.16.1.dev.

inprofitapi | 2020-06-10 09:51:42,228 - root - INFO - Initializing flask application...
inprofitapi | 2020-06-10 09:51:42,313 - root - INFO - Setting up kline subscription...
inprofitapi | 2020-06-10 09:51:42,315 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1.dev started ...
inprofitapi | 2020-06-10 09:51:42,593 - root - INFO - Starting missing candles loader...
inprofitapi | 2020-06-10 09:51:42,598 - root - INFO - Setting up mark price subscription...
inprofitapi | 2020-06-10 09:51:42,599 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1.dev started ...
inprofitapi | 2020-06-10 09:51:42,738 - root - INFO - Setting up mini ticker subscription...
inprofitapi | 2020-06-10 09:51:42,740 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1.dev started ...
inprofitapi | 2020-06-10 09:51:42,858 - root - INFO - Setting up user data subscription...
inprofitapi | 2020-06-10 09:51:42,886 - root - INFO - New instance of unicorn_binance_websocket_api_manager 1.16.1.dev started ...
inprofitapi | 2020-06-10 09:51:43,047 - root - INFO - Starting flask application...
oliver-zehentleitner commented 4 years ago

This fix is included now in 1.16.2 release!

oliver-zehentleitner commented 4 years ago

Can we close this issue? I think its successfully fixed.

Flowelfox commented 4 years ago

I think yes, no problems so far. Thank you for quick fix)

oliver-zehentleitner commented 4 years ago

Thanks for the good report, I think this fix made the lib stable, I was hunting this problem since a while :D (https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/issues/45)

oliver-zehentleitner commented 4 years ago

Its so weired. 8 days no issue, then I write its fixed and then it happens again: print_summary

4 streams stopped. I investigate it asap.

alekkhalex commented 3 years ago

I have same issue. A stream is crashing time after time:

ERROR:root:BinanceWebSocketApiSocket->start_socket(d1f4bcc1-092d-4e63-bbda-2003ec246d48, ['arr'], ['!userData']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError' CRITICAL:root:BinanceWebSocketApiManager->stream_is_crashing(d1f4bcc1-092d-4e63-bbda-2003ec246d48)

Is it possible to fix it?

Python 3.8.5 unicorn-binance-websocket-api 1.17.0 Windows 10

tahaEntesari commented 3 years ago

same issue here. using 1.16.9 with python 3.8.1 on windows 10

oliver-zehentleitner commented 3 years ago

I am not sure, but it seems the InvalidSateError happens if you have to less system powser (CPU).

Can you test it by making less streams or use a more powerful system?

tahaEntesari commented 3 years ago

I am running this program on a server and the cpu useage is typically low. It won't go beyong 50% no matter what happens. I have only two streams per user and these are the userData streams for SPOT and FUTURES At the moment there are 5 users that my program is keeping track of though. So, there are 5 different instances of websocket running in parallel each containing 2 streams. I don't think that I can personally test it. I mean, the program is under load and trading in binance and I can't take it down, at least not in a few days

oliver-zehentleitner commented 3 years ago

i am running different test systems of this lib since one year. on my systems i recognized even 50% on average is very much. sometimes are peaks that are too much then.

Running on a raspberry pi doenst work in general, dont know why.

You just need one instance per endpoint (spot/futures/...), doesnt matter how much users you have.

So your best setup should be two instances, one for spot and one for futures.

then you can create 5 spot userData streams within one instance and 5 futures userData streams.

you can split the stream_buffer, look at this example https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/blob/master/example_multiple_userdata_streams.py so you know which stream has received the data.

Till now you have had to use set_private_api_config() but its not thread safe and i added the possibility to add the key and secret to create_stream() see here: https://oliver-zehentleitner.github.io/unicorn-binance-websocket-api/unicorn_binance_websocket_api.html?highlight=create_stream#unicorn_binance_websocket_api.unicorn_binance_websocket_api_manager.BinanceWebSocketApiManager.create_stream

BUT: As you can see here https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/issues/109 the new way of providing key and secret doenst work now, i try to fix it this weekend.

oliver-zehentleitner commented 3 years ago

Its fixed providing key and secret via create_stream()!

Just update to 1.17.1

alekkhalex commented 3 years ago

Still facing same issue. A stream is crashing and reconnecting time after time:

ERROR:root:BinanceWebSocketApiSocket->start_socket(d1f4bcc1-092d-4e63-bbda-2003ec246d48, ['arr'], ['!userData']) Exception AttributeError Info: module 'asyncio.base_futures' has no attribute 'InvalidStateError' CRITICAL:root:BinanceWebSocketApiManager->stream_is_crashing(d1f4bcc1-092d-4e63-bbda-2003ec246d48)

Is it possible to fix it?

Python 3.8.5 unicorn-binance-websocket-api 1.17.1 Windows 2019 Server, 6 GB RAM, 4x 3.4ГГц CPU

oliver-zehentleitner commented 3 years ago

I will remove the InvalidStateError code part - it was a fix for an other problem. Lets make one step back.

tahaEntesari commented 3 years ago

@oliver-zehentleitner The odd thing is that this process is somewhat random. I mean, there are times that it happens like once every few hours but other times, its not that frequent. Since the stream is crashing each time, I think the websocket gets disconnected and thus any data sent in between the reconnect is lost (I have had inconsistent data only once in like 2 month but, I only use userData so I don't expect that much frequent data). So I think this is a somewhat important bug. One more thing. You added the parameters apiKey and apiSecret to create_stream but you have not added them to replace_stream(). I didn't check other places that it might have been useful.

alekkhalex commented 3 years ago

get_stream_info(stream_id) returns 'reconnects' = 1 after any qty of 'InvalidStateError' reconnects. Each time 'reconnects' = 1.

'logged_reconnects' = [..last reconnect timestamp..] - has only one element in the array of the last reconnection. May be it should be a list of timestamps of all reconnects?

Python 3.8.5 unicorn-binance-websocket-api 1.17.1 Windows 2019 Server

oliver-zehentleitner commented 3 years ago

@tahaEntesari Thanks for the hint about replace_stream() - i add it to the todo list. About disconnects in general: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/issues/42 But I am with you, we want it as stable as possible :)

@alekkhalex i recogniced that too, this bugs is new since 1.17.0 - also added to todo list.

Its much more handy if there is one issue for one topic. Please think about that in the future! And open new one if needed!

oliver-zehentleitner commented 3 years ago

InvalidStateError: https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/issues/110

oliver-zehentleitner commented 3 years ago

@tahaEntesar @alekkhalex

Both fixes are released now in 1.17.4

alekkhalex commented 3 years ago

Now all that reconnects have another message:

CRITICAL:root:BinanceWebSocketApiSocket->start_socket(09655d52-c65d-4ea3-9df7-5e242dc2577d, ['arr'], ['!userData']) Exception ConnectionClosed Info: code = 1006 (connection closed abnormally [internal]), no reason

From Binance documentation: -1006 UNEXPECTED_RESP An unexpected response was received from the message bus. Execution status unknown.

Python 3.8.5 unicorn-binance-websocket-api 1.17.4 Windows 2019 Server

oliver-zehentleitner commented 3 years ago

https://github.com/oliver-zehentleitner/unicorn-binance-websocket-api/issues/113