search

freqtrade / freqtrade

Free, open source crypto trading bot
https://www.freqtrade.io
GNU General Public License v3.0
28.15k stars 6.03k forks source link

Keyerror from API call #7215

Closed vaidab closed 2 years ago

vaidab commented 2 years ago

Describe your environment

Describe the problem:

I let my bot run in dry-run and today I saw that the strategy has a red circle which probably means it's offline and, when checking the logs, I see that it's filled with "Exception in ASGI application" errors. There's also a Telegram error, I've put it here but I don't know what is specifically the issue with the strategy not working.

Relevant code exceptions or logs


Traceback (most recent call last):
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/ext/updater.py", line 646, in _network_loop_retry
    if not action_cb():
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/ext/updater.py", line 597, in polling_action_cb
    updates = self.bot.get_updates(
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/ext/extbot.py", line 226, in get_updates
    updates = super().get_updates(
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/bot.py", line 133, in decorator
    result = func(*args, **kwargs)
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/bot.py", line 3057, in get_updates
    self._post(
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/bot.py", line 298, in _post
    return self.request.post(
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/utils/request.py", line 361, in post
    result = self._request_wrapper(
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/utils/request.py", line 265, in _request_wrapper
    raise NetworkError(f'urllib3 HTTPError {error}') from error
telegram.error.NetworkError: urllib3 HTTPError HTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /bot5530486623:AAEGgzLF0usor6znF2I7n4Jg9IuDMBES-Ig/getUpdates (Caused by NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.connection.VerifiedHTTPSConnection object at 0x7fba91ff46d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2022-08-11 14:40:59 - telegram.vendor.ptb_urllib3.urllib3.connectionpool - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.connection.VerifiedHTTPSConnection object at 0x7fba91ff7580>: Failed to establish a new connection: [Errno 111] Connection refused')': /bot5530486623:AAEGgzLF0usor6znF2I7n4Jg9IuDMBES-Ig/getUpdates
2022-08-11 14:40:59 - telegram.vendor.ptb_urllib3.urllib3.connectionpool - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.connection.VerifiedHTTPSConnection object at 0x7fba91ff6290>: Failed to establish a new connection: [Errno 111] Connection refused')': /bot5530486623:AAEGgzLF0usor6znF2I7n4Jg9IuDMBES-Ig/getUpdates
2022-08-11 14:40:59 - telegram.vendor.ptb_urllib3.urllib3.connectionpool - WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.connection.VerifiedHTTPSConnection object at 0x7fba91ff5ae0>: Failed to establish a new connection: [Errno 111] Connection refused')': /bot5530486623:AAEGgzLF0usor6znF2I7n4Jg9IuDMBES-Ig/getUpdates
2022-08-11 14:40:59 - telegram.ext.updater - ERROR - Error while getting Updates: urllib3 HTTPError HTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /bot5530486623:AAEGgzLF0usor6znF2I7n4Jg9IuDMBES-Ig/getUpdates (Caused by NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.connection.VerifiedHTTPSConnection object at 0x7fba91ff41f0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2022-08-11 14:40:59 - telegram.ext.dispatcher - ERROR - No error handlers are registered, logging exception.
Traceback (most recent call last):
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/connection.py", line 140, in _new_conn
    conn = connection.create_connection(
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/home/ftuser/.local/lib/python3.10/site-packages/telegram/vendor/ptb_urllib3/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

2022-08-11 16:35:17 - uvicorn.error - ERROR - Exception in ASGI application

Traceback (most recent call last):
  File "/home/ftuser/.local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/ftuser/.local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/home/ftuser/.local/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 86, in __call__
    raise exc from None
  File "/home/ftuser/.local/lib/python3.10/site-packages/uvicorn/middleware/message_logger.py", line 82, in __call__
    await self.app(scope, inner_receive, inner_send)
  File "/home/ftuser/.local/lib/python3.10/site-packages/fastapi/applications.py", line 269, in __call__
    await super().__call__(scope, receive, send)
  File "/home/ftuser/.local/lib/python3.10/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/ftuser/.local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/home/ftuser/.local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/home/ftuser/.local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/home/ftuser/.local/lib/python3.10/site-packages/starlette/exceptions.py", line 93, in __call__
    raise exc
  File "/home/ftuser/.local/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in __call__
    await self.app(scope, receive, sender)
  File "/home/ftuser/.local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/home/ftuser/.local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/home/ftuser/.local/lib/python3.10/site-packages/starlette/routing.py", line 670, in __call__
    await route.handle(scope, receive, send)
  File "/home/ftuser/.local/lib/python3.10/site-packages/starlette/routing.py", line 266, in handle
    await self.app(scope, receive, send)
  File "/home/ftuser/.local/lib/python3.10/site-packages/starlette/routing.py", line 65, in app
    response = await func(request)
  File "/home/ftuser/.local/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
    raw_response = await run_endpoint_function(
  File "/home/ftuser/.local/lib/python3.10/site-packages/fastapi/routing.py", line 162, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/home/ftuser/.local/lib/python3.10/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "/home/ftuser/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/ftuser/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/ftuser/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/freqtrade/freqtrade/rpc/api_server/api_v1.py", line 78, in profit
    return rpc._rpc_trade_statistics(config['stake_currency'],
  File "/freqtrade/freqtrade/rpc/rpc.py", line 445, in _rpc_trade_statistics
    current_rate = self._freqtrade.exchange.get_rate(
  File "/freqtrade/freqtrade/exchange/exchange.py", line 1544, in get_rate
    cache_rate[pair] = rate
  File "/home/ftuser/.local/lib/python3.10/site-packages/cachetools/ttl.py", line 87, in __setitem__
    self.expire(time)
  File "/home/ftuser/.local/lib/python3.10/site-packages/cachetools/ttl.py", line 167, in expire
    cache_delitem(self, curr.key)
  File "/home/ftuser/.local/lib/python3.10/site-packages/cachetools/cache.py", line 66, in __delitem__
    del self.__data[key]
KeyError: 'ZEN/USDT'  ```
xmatthias commented 2 years ago

You've got 2 errors here.

One is 2022-08-11 14:40:59 - telegram.vendor.ptb_urllib3.urllib3.connectionpool - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'NewConnectionError('<telegram.vendor.ptb_urllib3.urllib3.connection.VerifiedHTTPSConnection object at 0x7fba91ff6290>: Failed to establish a new connecti - which is an error connecting to the telegram API.

This most likely means that either their service was down (again) - or that there's been a problem with your network connection.

The "keyerror" is strange, as it comes from a location where we're writing the key, not reading it. It's also something i've never seen.

It shouldn't however be a critical error, and should result in this request failing - but nothing more.

I noticed that you're not running the latest version though - which makes investigating this difficult (things change all the time ...). Please update to the latest version (stable would be 2022.7) - and let me know if you can reproduce this there.

batuhanhazarguler commented 2 years ago 
2022-08-11 20:25:25 freqtrade.exchange.common: WARNING - get_tickers() returned exception: "Could not load tickers due to RequestTimeout. Message: binance GET https://fapi.binance.com/fapi/v1/ticker/24hr". Retrying still for 4 times.
2022-08-11 20:25:35 freqtrade.exchange.common: WARNING - get_tickers() returned exception: "Could not load tickers due to RequestTimeout. Message: binance GET https://fapi.binance.com/fapi/v1/ticker/24hr". Retrying still for 3 times.
2022-08-11 20:25:43 freqtrade.exchange.common: WARNING - get_tickers() returned exception: "Could not load tickers due to RequestTimeout. Message: binance GET https://fapi.binance.com/fapi/v1/ticker/24hr". Giving up.
2022-08-11 20:25:43 freqtrade.worker: WARNING - Error: Could not load tickers due to RequestTimeout. Message: binance GET https://fapi.binance.com/fapi/v1/ticker/24hr, retrying in 30 seconds...
2022-08-11 20:25:45 freqtrade.exchange.common: WARNING - get_tickers() returned exception: "Could not load tickers due to RequestTimeout. Message: binance GET https://fapi.binance.com/fapi/v1/ticker/24hr". Retrying still for 2 times.
2022-08-11 20:25:55 freqtrade.exchange.common: WARNING - get_tickers() returned exception: "Could not load tickers due to RequestTimeout. Message: binance GET https://fapi.binance.com/fapi/v1/ticker/24hr". Retrying still for 1 times.
2022-08-11 20:26:05 freqtrade.exchange.common: WARNING - get_tickers() returned exception: "Could not load tickers due to RequestTimeout. Message: binance GET https://fapi.binance.com/fapi/v1/ticker/24hr". Giving up.
2022-08-11 20:26:05 freqtrade.rpc.api_server.webserver: ERROR - API Error calling: Error getting current tickers.
NoneType: None
2022-08-11 20:26:13 freqtrade.worker: INFO - Bot heartbeat. PID=1, version='2022.7', state='RUNNING'
2022-08-11 20:26:23 freqtrade.exchange.common: WARNING - get_tickers() returned exception: "Could not load tickers due to RequestTimeout. Message: binance GET https://fapi.binance.com/fapi/v1/ticker/24hr". Retrying still for 4 times.

i am getting this error too is there any solution i need to do

emiliobasualdo commented 2 years ago

Same here. I don't think it is a problem with freqtrade guys.

Im placing the logs and stack trace to provide mode debbuging information Running on AmazonLinux AWS

2022-08-11 19:16:45,382 - freqtrade.exchange.exchange - ERROR - Could not reload markets.
Traceback (most recent call last):
  File "/home/ftuser/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/ftuser/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/ftuser/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/home/ftuser/.local/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/home/ftuser/.local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/home/ftuser/.local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ftuser/.local/lib/python3.8/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/home/ftuser/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/home/ftuser/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.binance.com', port=443): Max retries exceeded with url: /api/v3/exchangeInfo (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ftuser/.local/lib/python3.8/site-packages/ccxt/base/exchange.py", line 591, in fetch
    response = self.session.request(
  File "/home/ftuser/.local/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ftuser/.local/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/ftuser/.local/lib/python3.8/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.binance.com', port=443): Max retries exceeded with url: /api/v3/exchangeInfo (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/freqtrade/freqtrade/exchange/exchange.py", line 492, in reload_markets
    self._markets = self._api.load_markets(reload=True)
  File "/home/ftuser/.local/lib/python3.8/site-packages/ccxt/base/exchange.py", line 1445, in load_markets
    markets = self.fetch_markets(params)
  File "/home/ftuser/.local/lib/python3.8/site-packages/ccxt/binance.py", line 1448, in fetch_markets
    response = getattr(self, method)(query)
  File "/home/ftuser/.local/lib/python3.8/site-packages/ccxt/base/exchange.py", line 497, in inner
    return entry(_self, **inner_kwargs)
  File "/home/ftuser/.local/lib/python3.8/site-packages/ccxt/binance.py", line 5471, in request
    response = self.fetch2(path, api, method, params, headers, body, config, context)
  File "/home/ftuser/.local/lib/python3.8/site-packages/ccxt/base/exchange.py", line 2617, in fetch2
    return self.fetch(request['url'], request['method'], request['headers'], request['body'])
  File "/home/ftuser/.local/lib/python3.8/site-packages/ccxt/base/exchange.py", line 644, in fetch
    raise NetworkError(details) from e
ccxt.base.errors.NetworkError: binance GET https://api.binance.com/api/v3/exchangeInfo
2022-08-11 19:22:34,265 - freqtrade.exchange.common - WARNING - _async_get_candle_history() returned exception: "Could not fetch historical candle (OHLCV) data for pair BTC/USDT due to RequestTimeout. Message: binance GET https://api.binance.com/api/v3/klines?interval=1m&limit=1000&symbol=BTCUSDT". Retrying still for 4 times.
2022-08-11 19:22:45,013 - freqtrade.exchange.common - WARNING - _async_get_candle_history() returned exception: "Could not fetch historical candle (OHLCV) data for pair BTC/USDT due to RequestTimeout. Message: binance GET https://api.binance.com/api/v3/klines?interval=1m&limit=1000&symbol=BTCUSDT". Retrying still for 3 times.
2022-08-11 19:23:08,316 - freqtrade.exchange.common - WARNING - _async_get_candle_history() returned exception: "Could not fetch historical candle (OHLCV) data for pair BTC/USDT due to ExchangeNotAvailable. Message: binance GET https://api.binance.com/api/v3/klines?interval=1m&limit=1000&symbol=BTCUSDT". Retrying still for 2 times.
2022-08-11 19:23:21,015 - freqtrade.exchange.common - WARNING - _async_get_candle_history() returned exception: "Could not fetch historical candle (OHLCV) data for pair BTC/USDT due to RequestTimeout. Message: binance GET https://api.binance.com/api/v3/klines?interval=1m&limit=1000&symbol=BTCUSDT". Retrying still for 1 times.
2022-08-11 19:23:30,996 - freqtrade.exchange.common - WARNING - _async_get_candle_history() returned exception: "Could not fetch historical candle (OHLCV) data for pair BTC/USDT due to ExchangeNotAvailable. Message: binance GET https://api.binance.com/api/v3/klines?interval=1m&limit=1000&symbol=BTCUSDT". Giving up.
2022-08-11 19:23:31,246 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: TemporaryError('Could not fetch historical candle (OHLCV) data for pair BTC/USDT due to ExchangeNotAvailable. Message: binance GET https://api.binance.com/api/v3/klines?interval=1m&limit=1000&symbol=BTCUSDT')
2022-08-11 19:23:52,056 - freqtrade.strategy.interface - WARNING - Outdated history for pair BTC/USDT. Last tick is 41 minutes old
2022-08-11 19:23:53,217 - freqtrade.worker - INFO - Bot heartbeat. PID=1, version='2022.7', state='RUNNING'
xmatthias commented 2 years ago

@batuhanhazarguler @emiliobasualdo neither of your problems is in any way related to the problem reported here. As such, i've marked your comments as off-topic, as they're clearly not relevant to this issue.

Both of you have a Network error, which simply means that either your network connection had an outage, or binance had problems (potentially just in some area).

If the problem continues (and you're sure binance is up, reachable from your systems, please open a new issue, filling out the issue template - so it can be looked at separately.

xmatthias commented 2 years ago

Seems like caching is not fully threadsafe. should be fixed in develop (odd that it never appeared before ...).