search

bmoscon / cryptostore

A scalable storage service for cryptocurrency data
Other
384 stars 138 forks source link

Typo in feed naming? #189

Closed yohplala closed 3 years ago

yohplala commented 3 years ago

Describe the bug I have started cryptostore (last version of cryptofeed and cryptostore) about 2 days ago, and since about 12 hours, I have a continuously repeating warning & error, every 1s that says:

2021-07-08 19:34:39,383 : WARNING : BINANCE_FUTURES.http.6.http.6: received no messages within timeout, restarting connection
2021-07-08 19:34:39,961 : ERROR : BINANCE_FUTURES.http.6.http.6: encountered an exception, reconnecting in 1.0 seconds
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed/connection_handler.py", line 66, in _create_connection
    await self._handler(connection, self.handler)
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed/connection_handler.py", line 111, in _handler
    async for message in connection.read():
  File "/usr/local/lib/python3.8/dist-packages/cryptofeed/connection.py", line 164, in read
    async with self.conn.get(addr, headers=header) as response:
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/client.py", line 1117, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/client.py", line 520, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.8/dist-packages/aiohttp/connector.py", line 538, in connect
    raise ClientConnectionError("Connector is closed.")
aiohttp.client_exceptions.ClientConnectionError: Connector is closed.

I have other websocket connections for this server that work fine, and I have other servers in parallel, connecting to Binance for other pairs, with http and websocket which are working fine (not such warning and error repeating themsleves every 1s since 12 hours). I am surprised about this. I am about to restart the server with less pairs to see if this solves the trouble.

While I find this behavior suirprising, I think there might be a typo to correct in the warning and error. http.6 is repeated twice in ERROR : BINANCE_FUTURES.http.6.http.6. Is this normal? (for websocket connection, it is only written once, like ERROR : BINANCE_FUTURES.ws.1)

To Reproduce Steps to reproduce the behavior:

```yaml cache: redis redis: ip: '127.0.0.1' port: 6389 # port: 6379 socket: null del_after_read: true retention_time: null # start_flush: false start_flush: true exchanges: BINANCE: retries: -1 channel_timeouts: l2_book: 90 l2_book: symbols: ['ADA-ETH', 'ADA-BTC', 'ADA-USDT', 'ADA-BNB', 'ADA-USDC', 'ADA-BUSD', 'AKRO-BTC', 'AKRO-USDT', 'ALGO-BTC', 'ALGO-USDT', 'ALGO-BNB', 'ALGO-BUSD', 'ALPHA-BTC', 'ALPHA-USDT', 'ALPHA-BNB', 'ALPHA-BUSD', 'BCH-BTC', 'BCH-USDT', 'BCH-BNB', 'BCH-USDC', 'BCH-BUSD', 'BEL-BTC', 'BEL-USDT', 'BEL-BNB', 'BEL-BUSD', 'CELR-BTC', 'CELR-USDT', 'CELR-BNB', 'CELR-BUSD', 'CHZ-BTC', 'CHZ-USDT', 'CHZ-BNB', 'CHZ-BUSD', 'CVC-ETH', 'CVC-BTC', 'CVC-USDT', 'DENT-ETH', 'DENT-USDT', 'EGLD-BTC', 'EGLD-USDT', 'EGLD-BNB', 'EGLD-BUSD', 'ETH-BTC', 'ETH-USDT', 'ETH-DAI', 'ETH-USDC', 'ETH-PAX', 'ETH-BUSD', 'FTM-BTC', 'FTM-USDT', 'FTM-BNB', 'FTM-BUSD', 'ICP-BTC', 'ICP-USDT', 'ICP-BNB', 'ICP-BUSD', 'KNC-ETH', 'KNC-BTC', 'KNC-USDT', 'KNC-BUSD', 'LTC-ETH', 'LTC-BTC', 'LTC-USDT', 'LTC-BNB', 'LTC-USDC', 'LTC-BUSD', 'MTL-ETH', 'MTL-BTC', 'MTL-USDT', 'NEO-ETH', 'NEO-BTC', 'NEO-USDT', 'NEO-BNB', 'NEO-USDC', 'NEO-BUSD', 'RLC-ETH', 'RLC-BTC', 'RLC-USDT', 'RLC-BUSD', 'RSR-BTC', 'RSR-USDT', 'RSR-BNB', 'RSR-BUSD', 'SFP-BTC', 'SFP-USDT', 'SFP-BUSD', 'STMX-ETH', 'STMX-BTC', 'STMX-USDT', 'STMX-BNB', 'STMX-BUSD', 'SXP-BTC', 'SXP-USDT', 'SXP-BNB', 'SXP-BUSD', 'UNFI-BTC', 'UNFI-USDT', 'UNFI-BNB', 'UNFI-BUSD', 'XLM-ETH', 'XLM-BTC', 'XLM-USDT', 'XLM-BNB', 'XLM-BUSD', 'YFI-BTC', 'YFI-USDT', 'YFI-BNB', 'YFI-BUSD', 'ZEN-ETH', 'ZEN-BTC', 'ZEN-USDT', 'ZEN-BNB', 'ZEN-BUSD'] book_delta: true book_interval: 10000 BINANCE_FUTURES: retries: -1 channel_timeouts: l2_book: 90 funding: 90 open_interest: 90 l2_book: symbols: ['ADA-USDT', 'AKRO-USDT', 'ALGO-USDT', 'ALPHA-USDT', 'BCH-USDT', 'BEL-USDT', 'CELR-USDT', 'CHZ-USDT', 'CVC-USDT', 'DENT-USDT', 'EGLD-USDT', 'ETH-USDT', 'ETH-BUSD', 'FTM-USDT', 'ICP-USDT', 'KNC-USDT', 'LTC-USDT', 'MTL-USDT', 'NEO-USDT', 'RLC-USDT', 'RSR-USDT', 'SFP-USDT', 'STMX-USDT', 'SXP-USDT', 'UNFI-USDT', 'XLM-USDT', 'YFI-USDT', 'ZEN-USDT'] book_delta: true book_interval: 10000 funding: ['ADA-USDT', 'AKRO-USDT', 'ALGO-USDT', 'ALPHA-USDT', 'BCH-USDT', 'BEL-USDT', 'CELR-USDT', 'CHZ-USDT', 'CVC-USDT', 'DENT-USDT', 'EGLD-USDT', 'ETH-USDT', 'ETH-BUSD', 'FTM-USDT', 'ICP-USDT', 'KNC-USDT', 'LTC-USDT', 'MTL-USDT', 'NEO-USDT', 'RLC-USDT', 'RSR-USDT', 'SFP-USDT', 'STMX-USDT', 'SXP-USDT', 'UNFI-USDT', 'XLM-USDT', 'YFI-USDT', 'ZEN-USDT'] open_interest: ['ADA-USDT', 'AKRO-USDT', 'ALGO-USDT', 'ALPHA-USDT', 'BCH-USDT', 'BEL-USDT', 'CELR-USDT', 'CHZ-USDT', 'CVC-USDT', 'DENT-USDT', 'EGLD-USDT', 'ETH-USDT', 'ETH-BUSD', 'FTM-USDT', 'ICP-USDT', 'KNC-USDT', 'LTC-USDT', 'MTL-USDT', 'NEO-USDT', 'RLC-USDT', 'RSR-USDT', 'SFP-USDT', 'STMX-USDT', 'SXP-USDT', 'UNFI-USDT', 'XLM-USDT', 'YFI-USDT', 'ZEN-USDT'] BINANCE_DELIVERY: retries: -1 channel_timeouts: l2_book: 90 funding: 90 open_interest: 90 l2_book: symbols: ['ADA-USD_PERP', 'BCH-USD_PERP', 'EGLD-USD_PERP', 'ETH-USD_PERP', 'LTC-USD_PERP', 'XLM-USD_PERP'] book_delta: true book_interval: 10000 funding: ['ADA-USD_PERP', 'BCH-USD_PERP', 'EGLD-USD_PERP', 'ETH-USD_PERP', 'LTC-USD_PERP', 'XLM-USD_PERP'] open_interest: ['ADA-USD_PERP', 'BCH-USD_PERP', 'EGLD-USD_PERP', 'ETH-USD_PERP', 'LTC-USD_PERP', 'XLM-USD_PERP'] FTX: retries: -1 channel_timeouts: l2_book: 90 funding: 90 open_interest: 90 l2_book: symbols: ['ADA-PERP', 'AKRO-USD', 'AKRO-USDT', 'ALGO-PERP', 'ALPHA-USD', 'ALPHA-PERP', 'BCH-USD', 'BCH-BTC', 'BCH-USDT', 'BCH-PERP', 'CHZ-USD', 'CHZ-USDT', 'CHZ-PERP', 'DENT-USD', 'DENT-PERP', 'EGLD-PERP', 'ETH-USD', 'ETH-BTC', 'ETH-USDT', 'ETH-PERP', 'FTM-USD', 'FTM-PERP', 'ICP-PERP', 'KNC-USD', 'KNC-USDT', 'KNC-PERP', 'LTC-USD', 'LTC-BTC', 'LTC-USDT', 'LTC-PERP', 'MTL-USD', 'MTL-PERP', 'NEO-PERP', 'RSR-USD', 'RSR-PERP', 'STMX-USD', 'STMX-PERP', 'SXP-USD', 'SXP-BTC', 'SXP-USDT', 'SXP-PERP', 'XLM-PERP', 'YFI-USD', 'YFI-BTC', 'YFI-USDT', 'YFI-PERP'] book_delta: true book_interval: 10000 funding: ['ADA-PERP', 'AKRO-USD', 'AKRO-USDT', 'ALGO-PERP', 'ALPHA-USD', 'ALPHA-PERP', 'BCH-USD', 'BCH-BTC', 'BCH-USDT', 'BCH-PERP', 'CHZ-USD', 'CHZ-USDT', 'CHZ-PERP', 'DENT-USD', 'DENT-PERP', 'EGLD-PERP', 'ETH-USD', 'ETH-BTC', 'ETH-USDT', 'ETH-PERP', 'FTM-USD', 'FTM-PERP', 'ICP-PERP', 'KNC-USD', 'KNC-USDT', 'KNC-PERP', 'LTC-USD', 'LTC-BTC', 'LTC-USDT', 'LTC-PERP', 'MTL-USD', 'MTL-PERP', 'NEO-PERP', 'RSR-USD', 'RSR-PERP', 'STMX-USD', 'STMX-PERP', 'SXP-USD', 'SXP-BTC', 'SXP-USDT', 'SXP-PERP', 'XLM-PERP', 'YFI-USD', 'YFI-BTC', 'YFI-USDT', 'YFI-PERP'] open_interest: ['ADA-PERP', 'AKRO-USD', 'AKRO-USDT', 'ALGO-PERP', 'ALPHA-USD', 'ALPHA-PERP', 'BCH-USD', 'BCH-BTC', 'BCH-USDT', 'BCH-PERP', 'CHZ-USD', 'CHZ-USDT', 'CHZ-PERP', 'DENT-USD', 'DENT-PERP', 'EGLD-PERP', 'ETH-USD', 'ETH-BTC', 'ETH-USDT', 'ETH-PERP', 'FTM-USD', 'FTM-PERP', 'ICP-PERP', 'KNC-USD', 'KNC-USDT', 'KNC-PERP', 'LTC-USD', 'LTC-BTC', 'LTC-USDT', 'LTC-PERP', 'MTL-USD', 'MTL-PERP', 'NEO-PERP', 'RSR-USD', 'RSR-PERP', 'STMX-USD', 'STMX-PERP', 'SXP-USD', 'SXP-BTC', 'SXP-USDT', 'SXP-PERP', 'XLM-PERP', 'YFI-USD', 'YFI-BTC', 'YFI-USDT', 'YFI-PERP'] COINGECKO: retries: -1 channel_timeouts: market_info: 90 market_info: ['GRT-USD', 'SC-USD', 'REN-USD', 'STORJ-USD', 'AVAX-USD', 'SOL-USD', 'WAVES-USD', 'HBAR-USD', 'CVC-USD', 'DOGE-USD', 'UNI-USD', 'COTI-USD', 'BTS-USD', 'BCH-USD', 'CELR-USD', 'BAND-USD', 'CHR-USD', 'DENT-USD', 'USDC-USD', 'PAX-USD', 'BTC-USD', 'ZRX-USD', 'KEEP-USD', 'LIT-USD', 'BUSD-USD', 'ETH-USD', 'NEM-USD', 'REEF-USD', 'YFII-USD', 'SKALE-USD', 'OGN-USD', 'ICP-USD', 'OCEAN-USD', 'RLC-USD', 'UNFI-USD', 'KAVA-USD', 'RVN-USD', 'NKN-USD', 'RSR-USD', 'ADA-USD', 'LINA-USD', 'ZEC-USD', 'DASH-USD', 'HOT-USD', 'BTT-USD', 'ONT-USD', 'BZRX-USD', 'IOTA-USD', 'BAKE-USD', 'HNT-USD', 'USD-USD', 'TRX-USD', 'NEO-USD', 'CRV-USD', 'BNB-USD', 'ZEN-USD', 'VET-USD', 'KSM-USD', 'CTK-USD', 'YFI-USD', 'EGLD-USD', 'MTL-USD', 'DGB-USD', 'LRC-USD', 'ANKR-USD', 'XLM-USD', 'BAT-USD', 'SFP-USD', 'RUNE-USD', '1INCH-USD', 'FLM-USD', 'ATOM-USD', 'XTZ-USD', 'LTC-USD', 'SUSHI-USD', 'USDT-USD', 'MANA-USD', 'PERP-USD', 'ICON-USD', 'EOS-USD', 'AXS-USD', 'DODO-USD', 'TRB-USD', 'SXP-USD', 'OMG-USD', 'IOST-USD', 'ALGO-USD', 'NEAR-USD', 'AKRO-USD', 'QTUM-USD', 'CHZ-USD', 'STMX-USD', 'TOMO-USD', 'BLZ-USD', 'DOT-USD', 'SRM-USD', 'THETA-USD', 'MKR-USD', 'ZIL-USD', 'AAVE-USD', 'BEL-USD', 'ETC-USD', 'MATIC-USD', 'SNX-USD', 'KNC-USD', 'ALICE-USD', 'FTM-USD', 'ENJ-USD', 'FIL-USD', 'BAL-USD', 'COMP-USD', 'ALPHA-USD', 'LINK-USD', 'XMR-USD', 'GTC-USD', 'DAI-USD', 'XRP-USD'] storage: [parquet] parquet: # del_file: true append_counter: 29 file_format: [exchange, data_type, symbol, timestamp] compression: codec: 'BROTLI' level: 3 path: '/root/yohplala/data' prefix_date: false storage_interval: 2M storage_retries: 5 storage_retry_wait: 30 ```

Operating System:

Cryptofeed Version fresh from source 2 days ago.

yohplala commented 3 years ago

Hi, I have restarted a second server and am about to stop that one onto which are being issued these repeating error/warning. I am loosely monitoring CPU and RAM activity with htop, and since these error/warning started, the RAM has been increasing. (from about ~600Mo to ~800Mo in more than 1 day) I have no clear data to share, and I am suspecting that the channel is still open, and is filling in redis buffer (the http stream is likely funding from BINANCE_FUTURES), but somehow cryptofeed 'lost the handle'. Is this possible?

(I may be completely wrong of course)

bmoscon commented 3 years ago

I fixed this issue in cryptofeed