search

LUCIT-Systems-and-Development / unicorn-binance-websocket-api

A Python SDK by LUCIT to use the Binance Websocket API`s (com+testnet, com-margin+testnet, com-isolated_margin+testnet, com-futures+testnet, com-coin_futures, us, tr, dex/chain+testnet) in a simple, fast, flexible, robust and fully-featured way.
https://unicorn-binance-websocket-api.docs.lucit.tech/
Other
691 stars 164 forks source link

Socket timeout and thread exception using unicorn-binance-websocket-api 1.41.0 #274

Closed botradingblog1 closed 1 year ago

botradingblog1 commented 2 years ago

Version of this library.

1.41.0

Solution to Issue cannot be found in the documentation or other Issues and also occurs in the latest version of this library.

Hardware?

Local server/workstation

Operating System?

Windows

Python version?

Python3.9

Installed packages

absl-py                       0.15.0
aiohttp                       3.7.4
alpaca-trade-api              1.5.0
aniso8601                     9.0.1
appdirs                       1.4.4
astunparse                    1.6.3
async-generator               1.10
async-timeout                 3.0.1
attrs                         21.4.0
backcall                      0.2.0
beautifulsoup4                4.9.3
bpemb                         0.3.3
bs4                           0.0.1
cachetools                    5.0.0
certifi                       2021.10.8
cffi                          1.15.0
chardet                       3.0.4
charset-normalizer            2.0.10
cheroot                       8.6.0
clang                         5.0
click                         8.0.3
cmdstanpy                     0.9.68
colorama                      0.4.4
commonmark                    0.9.1
conllu                        4.4.1
convertdate                   2.4.0
cryptography                  37.0.1
cssselect                     1.1.0
cycler                        0.11.0
Cython                        0.29.23
dateparser                    0.7.6
debugpy                       1.5.1
decorator                     5.1.1
Deprecated                    1.2.13
deprecation                   2.1.0
dnspython                     1.16.0
entrypoints                   0.3
ephem                         4.1.3
fake-useragent                0.1.11
feedparser                    6.0.8
filelock                      3.4.2
fix-yahoo-finance             0.1.37
flair                         0.10
Flask                         2.1.1
Flask-RESTful                 0.3.9
flatbuffers                   1.12
fonttools                     4.29.0
ftfy                          6.0.3
gast                          0.4.0
gdown                         3.12.2
gensim                        4.1.2
gnews                         0.2.7
google-auth                   2.5.0
google-auth-oauthlib          0.4.6
google-pasta                  0.2.0
grpcio                        1.43.0
h11                           0.13.0
h5py                          3.1.0
hijri-converter               2.2.2
holidays                      0.12
huggingface-hub               0.4.0
idna                          3.3
importlib-metadata            4.10.1
ipykernel                     6.4.1
ipython                       7.31.1
ipython-genutils              0.2.0
itsdangerous                  2.1.2
Janome                        0.4.1
jaraco.functools              3.5.0
jedi                          0.18.0
Jinja2                        3.1.1
joblib                        1.1.0
jsonschema                    4.4.0
jupyter-client                7.1.2
jupyter-core                  4.9.1
kaleido                       0.1.0.post1
keras                         2.8.0
Keras-Preprocessing           1.1.2
keras-tuner                   1.1.0
keybert                       0.5.0
kiwisolver                    1.3.2
konoha                        4.6.5
korean-lunar-calendar         0.2.1
kt-legacy                     1.0.4
langdetect                    1.0.9
libclang                      13.0.0
loguru                        0.5.3
LunarCalendar                 0.0.9
lxml                          4.7.1
Markdown                      3.3.6
MarkupSafe                    2.1.1
matplotlib                    3.5.1
matplotlib-inline             0.1.2
more-itertools                8.8.0
mpld3                         0.3
mplfinance                    0.12.8b9
msgpack                       1.0.2
multidict                     6.0.2
multitasking                  0.0.10
munkres                       1.1.4
nbformat                      5.1.3
nest-asyncio                  1.5.1
nltk                          3.6.7
numpy                         1.22.1
oauthlib                      3.1.1
olefile                       0.46
opt-einsum                    3.3.0
outcome                       1.1.0
overrides                     3.1.0
packaging                     21.3
pandas                        1.4.0
pandas-datareader             0.10.0
pandas-ta                     0.3.14b0
parse                         1.19.0
parso                         0.8.3
patsy                         0.5.2
pickleshare                   0.7.5
Pillow                        9.0.0
pip                           21.2.4
plotly                        5.6.0
plotly-express                0.4.1
praw                          7.5.0
prawcore                      2.3.0
prompt-toolkit                3.0.20
prophet                       1.0.1
protobuf                      3.19.3
psutil                        5.9.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycparser                     2.21
pydot                         1.4.2
pyee                          8.2.2
Pygments                      2.10.0
PyMeeus                       0.5.11
pymongo                       3.12.3
pyOpenSSL                     22.0.0
pyparsing                     3.0.7
pyppeteer                     1.0.2
PyQt5                         5.12.3
PyQt5_sip                     4.19.18
PyQtChart                     5.12
PyQtWebEngine                 5.12.1
pyquery                       1.4.3
pyrsistent                    0.18.1
PySocks                       1.7.1
pystan                        2.19.1.1
python-binance                1.0.16
python-dateutil               2.8.2
python-dotenv                 0.19.2
python-engineio               3.14.2
python-socketio               4.6.0
pytz                          2021.3
pytz-deprecation-shim         0.1.0.post0
pywin32                       302
PyYAML                        5.4.1
pyzmq                         22.3.0
regex                         2022.1.18
requests                      2.26.0
requests-html                 0.10.0
requests-oauthlib             1.3.0
rich                          11.2.0
rsa                           4.8
sacremoses                    0.0.47
scikit-learn                  1.0.2
scipy                         1.7.3
seaborn                       0.11.2
segtok                        1.5.11
selenium                      4.1.3
sentence-transformers         2.2.0
sentencepiece                 0.1.95
setuptools                    57.5.0
setuptools-git                1.2
sgmllib3k                     1.0.0
simfin                        0.8.4
six                           1.16.0
smart-open                    5.2.1
sniffio                       1.2.0
sortedcontainers              2.4.0
soupsieve                     2.3.1
sqlitedict                    1.7.0
statsmodels                   0.13.1
ta                            0.9.0
TA-Lib                        0.4.24
tabulate                      0.8.9
tenacity                      8.0.1
tensorboard                   2.8.0
tensorboard-data-server       0.6.1
tensorboard-plugin-wit        1.8.1
tensorflow                    2.6.0
tensorflow-estimator          2.7.0
tensorflow-gpu                2.8.0
tensorflow-io-gcs-filesystem  0.23.1
termcolor                     1.1.0
tf-estimator-nightly          2.8.0.dev2021122109
threadpoolctl                 3.0.0
tokenizers                    0.10.3
torch                         1.10.2
torchvision                   0.11.3
tornado                       6.1
tqdm                          4.62.3
traitlets                     5.1.1
transformers                  4.15.0
trio                          0.20.0
trio-websocket                0.9.2
tweepy                        4.5.0
typing-extensions             3.7.4.3
tzdata                        2022.1
tzlocal                       4.1
ujson                         5.1.0
unicodedata2                  14.0.0
unicorn-binance-websocket-api 1.41.0
unicorn-fy                    0.12.2
update-checker                0.18.0
urllib3                       1.26.8
uuid                          1.30
v                             0.0.0
w3lib                         1.22.0
wcwidth                       0.2.5
websocket-client              1.2.3
websockets                    10.3
Werkzeug                      2.0.2
wheel                         0.37.1
Wikipedia-API                 0.5.4
win32-setctime                1.1.0
wincertstore                  0.2
wrapt                         1.12.1
wsproto                       1.1.0
yahoo-fin                     0.8.9.1
yarl                          1.7.2
yfinance                      0.1.70
zipp                          3.7.0

Logging output

Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\urllib3\connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\urllib3\connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\http\client.py", line 1345, in getresponse
    response.begin()
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\http\client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\http\client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\urllib3\connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\urllib3\util\retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\urllib3\packages\six.py", line 770, in reraise
    raise value
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\urllib3\connectionpool.py", line 451, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\urllib3\connectionpool.py", line 340, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='testnet.binance.vision', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\threading.py", line 954, in _bootstrap_inner
    self.run()
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "C:\dev\trading\tradesystem1\crypto-bots\binance_trading_bot2.py", line 510, in process_stream_data
    handle_price_change(symbol=data['s'], timestamp=data['T'], price=float(data['p']))
  File "C:\dev\trading\tradesystem1\crypto-bots\binance_trading_bot2.py", line 401, in handle_price_change
    historic_prices_df = get_crypto_minute_prices(symbol)
  File "C:\dev\trading\tradesystem1\crypto-bots\binance_trading_bot2.py", line 642, in get_crypto_minute_prices
    df = pd.DataFrame(client.get_historical_klines(symbol, '1m', '300 minutes ago UTC'))
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\binance\client.py", line 934, in get_historical_klines
    return self._historical_klines(
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\binance\client.py", line 973, in _historical_klines
    first_valid_ts = self._get_earliest_valid_timestamp(symbol, interval, klines_type)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\binance\client.py", line 904, in _get_earliest_valid_timestamp
    kline = self._klines(
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\binance\client.py", line 883, in _klines
    return self.get_klines(**params)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\binance\client.py", line 868, in get_klines
    return self._get('klines', data=params, version=self.PRIVATE_API_VERSION)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\binance\client.py", line 371, in _get
    return self._request_api('get', path, signed, version, **kwargs)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\binance\client.py", line 334, in _request_api
    return self._request(method, uri, signed, **kwargs)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\binance\client.py", line 314, in _request
    self.response = getattr(self.session, method)(uri, **kwargs)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\requests\sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\dutch\anaconda3\envs\tradesystem1\lib\site-packages\requests\adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='testnet.binance.vision', port=443): Read timed out. (read timeout=10)
BinanceWebSocketApiSocket.start_socket(154321d1a832-0ec9-db58-20d6-1d999650, {'trade'}, ['btcusdt']) - Exception ConnectionClosed - error_msg: sent 1011 (unexpected error) keepalive ping timeout; no close frame received
BinanceWebSocketApiManager.stream_is_crashing(154321d1a832-0ec9-db58-20d6-1d999650)
BinanceWebSocketApiManager._create_stream_thread() stream_id=154321d1a832-0ec9-db58-20d6-1d999650  - RuntimeError `error: 11` - error_msg:  cannot schedule new futures after interpreter shutdown - Please create an issue: https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/issues/new/choose

Processing method?

process_stream_data

Used endpoint?

binance.us

Issue

This happens with binance-us testnet. I ran the websocket connection for about 5 hours when the issue occurred. It may just be an issue on the Binance side. Can you tell me how I could handle this exception and restart the websocket listener? Thanks!

oliver-zehentleitner commented 2 years ago

Please post your code!

I guess you have only one stream and are using high_performance=True? Try with setting to False!

botradingblog1 commented 2 years ago

Hi, thanks for the feedback. I'm not explicitly setting high_performance.

Here my code:

from unicorn_binance_websocket_api.manager import BinanceWebSocketApiManager

binance_us_websocket_api_manager = BinanceWebSocketApiManager(exchange="binance.us")

def process_stream_data(binance_websocket_api_manager):
    while True:
        if binance_websocket_api_manager.is_manager_stopping():
            exit(0)
        oldest_data = binance_websocket_api_manager.pop_stream_data_from_stream_buffer()

        is_empty = is_empty_message(oldest_data)
        if is_empty:
            time.sleep(0.02)
        else:
            oldest_data_dict = json.loads(oldest_data)
            data = oldest_data_dict['data']

            #  Handle price change
            handle_price_change(symbol=data['s'], timestamp=data['T'], price=float(data['p']))

def start_websocket_listener():
    global stop_running
    channels = {'trade', }

    try:
        binance_us_websocket_api_manager.create_stream(channels, markets=lc_symbols, api_key=TEST_API_KEY, api_secret=TEST_API_SECRET)
        # Start a worker process to move the received stream_data from the stream_buffer to a print function
        worker_thread = threading.Thread(target=process_stream_data, args=(binance_us_websocket_api_manager,))
        worker_thread.start()
    except Exception as e:
        logger.error('Failure in worker thread for websocket: ', e)
oliver-zehentleitner commented 2 years ago

I dont see the reason sorry.

I need to test this, but it will need a while, i am very busy with other stuff at the moment.

oliver-zehentleitner commented 2 years ago

You mix api test key with binance us? I think that is in general not supported?