search

LUCIT-Systems-and-Development / unicorn-binance-local-depth-cache

A Python SDK by LUCIT for accessing and managing multiple local Binance order books with Python in a simple, fast, flexible, robust and fully featured way.
https://unicorn-binance-local-depth-cache.docs.lucit.tech/
Other
36 stars 10 forks source link

BinanceLocalDepthCacheManager._init_depth_cache() - Can not download order_book snapshot for the depth_cache #27

Closed foentiex closed 2 years ago

foentiex commented 2 years ago

Version of this library.

0.7.2

Solution to Issue cannot be found in the documentation or other Issues and also occurs in the latest version of this library.

Hardware?

Local server/workstation

Operating System?

Linux

Python version?

Python3.9

Installed packages

Package                                   Version
----------------------------------------- -------------------
absl-py                                   0.15.0
aniso8601                                 9.0.1
argon2-cffi                               21.3.0
argon2-cffi-bindings                      21.2.0
asttokens                                 2.0.5
astunparse                                1.6.3
asyncio                                   3.4.3
attrs                                     21.2.0
auto-sklearn                              0.14.5
autokeras                                 1.0.16.post1
backcall                                  0.2.0
black                                     22.1.0
bleach                                    4.1.0
cachetools                                5.0.0
certifi                                   2021.10.8
cffi                                      1.15.0
charset-normalizer                        2.0.8
cheroot                                   8.5.2
click                                     8.0.3
cloudpickle                               2.0.0
colorama                                  0.4.4
configparser                              5.1.0
ConfigSpace                               0.4.20
cryptography                              36.0.0
cycler                                    0.11.0
Cython                                    0.29.27
dask                                      2022.1.0
dateparser                                1.1.0
debugpy                                   1.5.1
decorator                                 5.1.1
defusedxml                                0.7.1
distributed                               2022.1.0
distro                                    1.6.0
emcee                                     3.1.1
entrypoints                               0.3
eventhandler                              1.2.1
executing                                 0.8.2
FLAML                                     0.9.5
Flask                                     2.0.2
Flask-RESTful                             0.3.9
flatbuffers                               1.12
fonttools                                 4.28.5
fsspec                                    2022.1.0
gast                                      0.4.0
google-auth                               2.6.0
google-auth-oauthlib                      0.4.6
google-pasta                              0.2.0
grpcio                                    1.34.1
h5py                                      3.1.0
HeapDict                                  1.0.1
idna                                      3.3
importlib-metadata                        4.10.1
investpy                                  1.0.7
ipykernel                                 6.7.0
ipython                                   7.31.1
ipython-genutils                          0.2.0
itsdangerous                              2.0.1
jaraco.functools                          3.4.0
jedi                                      0.18.1
Jinja2                                    3.0.3
joblib                                    1.1.0
jsonschema                                4.4.0
jupyter-client                            7.1.2
jupyter-core                              4.9.1
jupyterlab-pygments                       0.1.2
keras-nightly                             2.5.0.dev2021032900
Keras-Preprocessing                       1.1.2
keras-tuner                               1.0.4
kiwisolver                                1.3.2
kt-legacy                                 1.0.4
liac-arff                                 2.5.0
lightgbm                                  3.3.2
llvmlite                                  0.38.0
locket                                    0.2.1
lxml                                      4.6.4
Markdown                                  3.3.6
MarkupSafe                                2.0.1
matplotlib                                3.5.1
matplotlib-inline                         0.1.3
matrixprofile                             1.1.10
mistune                                   0.8.4
more-itertools                            8.12.0
msgpack                                   1.0.3
mypy-extensions                           0.4.3
nbclient                                  0.5.10
nbconvert                                 6.4.1
nbformat                                  5.1.3
nest-asyncio                              1.5.4
networkx                                  2.6.3
notebook                                  6.4.8
numba                                     0.55.0
numpy                                     1.21.5
oauthlib                                  3.2.0
opt-einsum                                3.3.0
packaging                                 21.3
pandas                                    1.3.4
pandocfilters                             1.5.0
parso                                     0.8.3
partd                                     1.2.0
pathlib                                   1.0.1
pathspec                                  0.9.0
patsy                                     0.5.2
pexpect                                   4.8.0
pickleshare                               0.7.5
pika                                      1.2.0
pika-stubs                                0.1.3
Pillow                                    9.0.0
pip                                       22.1
pipelineprofiler                          0.1.17
platformdirs                              2.4.1
prometheus-client                         0.13.1
prompt-toolkit                            3.0.26
protobuf                                  3.19.4
psutil                                    5.8.0
ptyprocess                                0.7.0
pure-eval                                 0.2.2
pyaes                                     1.6.1
pyasn1                                    0.4.8
pyasn1-modules                            0.2.8
pycparser                                 2.21
Pygments                                  2.11.2
pynisher                                  0.6.4
pyOpenSSL                                 21.0.0
pyparsing                                 3.0.7
pyproj                                    3.3.0
pyrfr                                     0.8.2
pyrsistent                                0.18.1
pyshp                                     2.2.0
pysqlite3                                 0.4.6
python-dateutil                           2.8.2
pytz                                      2021.3
pytz-deprecation-shim                     0.1.0.post0
PyYAML                                    6.0
pyzmq                                     22.3.0
regex                                     2021.11.10
requests                                  2.26.0
requests-oauthlib                         1.3.1
rsa                                       4.8
scikit-learn                              0.24.2
scipy                                     1.7.3
Send2Trash                                1.8.0
service-identity                          21.1.0
setuptools                                59.4.0
simdkalman                                1.0.2
six                                       1.15.0
smac                                      1.1.1
sortedcontainers                          2.4.0
spark                                     0.2.1
stack-data                                0.1.4
statsmodels                               0.13.1
stockstats                                0.4.1
stumpy                                    1.10.2
talipp                                    1.7.0
tblib                                     1.7.0
Telethon                                  1.24.0
tensorboard                               2.8.0
tensorboard-data-server                   0.6.1
tensorboard-plugin-wit                    1.8.1
tensorflow                                2.5.2
tensorflow-estimator                      2.5.0
termcolor                                 1.1.0
terminado                                 0.13.1
testpath                                  0.5.0
threadpoolctl                             3.0.0
tomli                                     2.0.0
toolz                                     0.11.2
tornado                                   6.1
tqdm                                      4.62.3
traitlets                                 5.1.1
tsfresh                                   0.19.0
tsmoothie                                 1.0.4
typing-extensions                         3.7.4.3
tzdata                                    2021.5
tzlocal                                   4.1
ujson                                     4.3.0
unicorn-binance-local-depth-cache         0.7.2
unicorn-binance-rest-api                  1.5.0
unicorn-binance-trailing-stop-loss-engine 0.0.0
unicorn-binance-websocket-api             1.41.0
unicorn-fy                                0.11.0
Unidecode                                 1.3.2
urllib3                                   1.26.7
wcwidth                                   0.2.5
webencodings                              0.5.1
websocket-client                          1.2.1
websockets                                10.3
Werkzeug                                  2.0.2
wheel                                     0.37.0
wrapt                                     1.12.1
xgboost                                   1.3.3
zict                                      2.0.0
zipp                                      3.7.0

Logging output

Sorry, don't know where is the log file

Used endpoint?

binance.com-futures

Issue

Hello. Testing the library with this code:

-------------------------------------------------------

future_client = BinanceRestApiManager(exchange="binance.com-futures", api_key=api_key, api_secret=api_secret)

bw_api_manager = BinanceWebSocketApiManager(exchange="binance.com-futures", high_performance=True, enable_stream_signal_buffer=True) ubldc = BinanceLocalDepthCacheManager(exchange="binance.com-futures", ubwa_manager=bw_api_manager) markets = ['BNBBUSD', 'ETHBUSD', 'GMTBUSD', 'XRPBUSD', 'SOLBUSD', 'ADABUSD', 'DOGEBUSD'] ubldc.create_depth_cache(markets=markets, refresh_interval=5)

while True: stream_data = bw_api_manager.pop_stream_data_from_stream_buffer() kline = json.loads(stream_data) symbol = kline['data']['s']

# doing things with the stream

if condition(symbol): # This condition is True more or less 6 time in 10',  in less than a 10'' 
       asks = ubldc.get_asks(market=market)[:10]
       bids = ubldc.get_bids(market=market)[:10]

--------------

Before an hour the next error is showing:

BinanceLocalDepthCacheManager._init_depth_cache() - Can not download order_book snapshot for the depth_cache with market ethbusd - requests.exceptions.ReadTimeout - error_msg: HTTPSConnectionPool(host='fapi.binance.com', port=443): Read timed out. (read timeout=10) Exception in thread Thread-5: Traceback (most recent call last): File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner self.run() File "/usr/lib/python3.9/threading.py", line 910, in run self._target(*self._args, **self._kwargs) File "/home/workspace/crypto/futuress/venv/lib/python3.9/site-packages/unicorn_binance_local_depth_cache/manager.py", line 372, in _process_stream_data if not self._init_depth_cache(market=market): File "/home/workspace/crypto/futures/venv/lib/python3.9/site-packages/unicorn_binance_local_depth_cache/manager.py", line 326, in _init_depth_cache self.depth_caches[market]['last_update_id'] = int(order_book['lastUpdateId']) TypeError: 'NoneType' object is not subscriptable

oliver-zehentleitner commented 2 years ago

Hello!

Do you get this error everytime you start it or just sometimes?

I can provide a fix to handle this error and reinit the cache.

foentiex commented 2 years ago

Hello. Today it has been three times that has given the error. Now it takes an hour without failure

oliver-zehentleitner commented 2 years ago

Ok, i will post an update asap. I think next week

oliver-zehentleitner commented 2 years ago

In the meantime you could build a try-except block arround the depthcache command and excpet TypeError and just restart the manager

foentiex commented 2 years ago

Ok, thank you

oliver-zehentleitner commented 2 months ago

2.0.0 has been released https://pypi.org/project/unicorn-binance-local-depth-cache/