search

bmoscon / cryptostore

A scalable storage service for cryptocurrency data
Other
383 stars 138 forks source link

Redis + Parquet -> AWS S3 ERROR #80

Closed xiandong79 closed 4 years ago

xiandong79 commented 4 years ago 
2020-04-15 03:51:41,023 : ERROR : KRAKEN: encountered an exception, reconnecting
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 208, in _connect
    await self._handler(websocket, feed.message_handler, feed.uuid)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 237, in _handler
    await handler(message, self.last_msg[feed_id])
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/exchange/kraken.py", line 131, in message_handler
    await self._book(msg, self.channel_map[msg[0]][1], timestamp)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/exchange/kraken.py", line 96, in _book
    await self.book_callback(self.l2_book[pair], L2_BOOK, pair, True, delta, timestamp, timestamp)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feed.py", line 106, in book_callback
    await self.callback(L2_BOOK, feed=self.id, pair=pair, book=book, timestamp=timestamp, receipt_timestamp=receipt_timestamp)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feed.py", line 113, in callback
    await cb(**kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/backend.py", line 17, in __call__
    await self.write(feed, pair, timestamp, receipt_timestamp, data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/redis.py", line 71, in write
    await super().write(feed, pair, timestamp, receipt_timestamp, data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/redis.py", line 39, in write
    await self.redis.xadd(f"{self.key}-{feed}-{pair}", data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/pool.py", line 257, in _wait_execute
    conn = await self.acquire(command, args)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/pool.py", line 324, in acquire
    await self._fill_free(override_min=True)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/pool.py", line 386, in _fill_free
    await conn.execute('ping')
aioredis.errors.ReplyError: LOADING Redis is loading the dataset in memory
2020-04-15 03:51:41,209 : ERROR : COINBASE: encountered an exception, reconnecting
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 208, in _connect
    await self._handler(websocket, feed.message_handler, feed.uuid)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 237, in _handler
    await handler(message, self.last_msg[feed_id])
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/exchange/coinbase.py", line 282, in message_handler
    await self._book_update(msg, timestamp)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/exchange/coinbase.py", line 126, in _book_update
    receipt_timestamp=timestamp
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feed.py", line 113, in callback
    await cb(**kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/backend.py", line 33, in __call__
    await self.write(feed, pair, timestamp, receipt_timestamp, data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/redis.py", line 39, in write
    await self.redis.xadd(f"{self.key}-{feed}-{pair}", data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/pool.py", line 257, in _wait_execute
    conn = await self.acquire(command, args)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/pool.py", line 324, in acquire
    await self._fill_free(override_min=True)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/pool.py", line 386, in _fill_free
    await conn.execute('ping')
aioredis.errors.ReplyError: LOADING Redis is loading the dataset in memory

2020-04-15 03:51:36,605 : WARNING : BYBIT: encountered connection issue [Errno 104] Connection reset by peer - reconnecting...
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 208, in _connect
    await self._handler(websocket, feed.message_handler, feed.uuid)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 237, in _handler
    await handler(message, self.last_msg[feed_id])
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/exchange/bybit.py", line 41, in message_handler
    await self._book(msg, timestamp)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/exchange/bybit.py", line 113, in _book
    await self.book_callback(self.l2_book[pair], L2_BOOK, pair, forced, delta, msg['timestamp_e6'] / 1000000, timestamp)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feed.py", line 106, in book_callback
    await self.callback(L2_BOOK, feed=self.id, pair=pair, book=book, timestamp=timestamp, receipt_timestamp=receipt_timestamp)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feed.py", line 113, in callback
    await cb(**kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/backend.py", line 17, in __call__
    await self.write(feed, pair, timestamp, receipt_timestamp, data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/redis.py", line 71, in write
    await super().write(feed, pair, timestamp, receipt_timestamp, data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/redis.py", line 39, in write
    await self.redis.xadd(f"{self.key}-{feed}-{pair}", data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/connection.py", line 186, in _read_data
    obj = await self._reader.readobj()
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/stream.py", line 102, in readobj
    await self._wait_for_data('readobj')
  File "/usr/lib/python3.7/asyncio/streams.py", line 473, in _wait_for_data
    await self._waiter
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 814, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
2020-04-15 03:51:36,611 : WARNING : BYBIT: encountered connection issue [Errno 104] Connection reset by peer - reconnecting...
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 208, in _connect
    await self._handler(websocket, feed.message_handler, feed.uuid)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 237, in _handler
    await handler(message, self.last_msg[feed_id])
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/exchange/bybit.py", line 39, in message_handler
    await self._trade(msg, timestamp)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/exchange/bybit.py", line 81, in _trade
    receipt_timestamp=timestamp
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feed.py", line 113, in callback
    await cb(**kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/backend.py", line 33, in __call__
    await self.write(feed, pair, timestamp, receipt_timestamp, data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/redis.py", line 39, in write
    await self.redis.xadd(f"{self.key}-{feed}-{pair}", data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/connection.py", line 186, in _read_data
    obj = await self._reader.readobj()
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/stream.py", line 102, in readobj
    await self._wait_for_data('readobj')
  File "/usr/lib/python3.7/asyncio/streams.py", line 473, in _wait_for_data
    await self._waiter
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 814, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
2020-04-15 03:51:36,642 : WARNING : BINANCE: encountered connection issue [Errno 104] Connection reset by peer - reconnecting...
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 208, in _connect
    await self._handler(websocket, feed.message_handler, feed.uuid)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 237, in _handler
    await handler(message, self.last_msg[feed_id])
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/exchange/binance.py", line 211, in message_handler
    await self._trade(msg, timestamp)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/exchange/binance.py", line 73, in _trade
    receipt_timestamp=timestamp)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/feed.py", line 113, in callback
    await cb(**kwargs)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/backend.py", line 33, in __call__
    await self.write(feed, pair, timestamp, receipt_timestamp, data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/cryptofeed/backends/redis.py", line 39, in write
    await self.redis.xadd(f"{self.key}-{feed}-{pair}", data)
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/connection.py", line 186, in _read_data
    obj = await self._reader.readobj()
  File "/home/ubuntu/.local/lib/python3.7/site-packages/aioredis/stream.py", line 102, in readobj
    await self._wait_for_data('readobj')
  File "/usr/lib/python3.7/asyncio/streams.py", line 473, in _wait_for_data
    await self._waiter
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 814, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
cache: redis

redis:
    ip: '127.0.0.1'
    port: 6379
    socket: null
    del_after_read: true
    retention_time: null
    start_flush: true

exchanges:
    BITMEX:
        retries: 30
        l2_book:
            symbols: [XBTUSD]
            book_interval: 600
        trades: [XBTUSD]
    KRAKEN:
        retries: 30
        l2_book:
            symbols: [BTC-USD]
            max_depth: 20
            book_interval: 600
        trades: [BTC-USD]
    BITFINEX:
        retries: 30
        l2_book:
            symbols: [BTC-USD, USDT-USD]
            max_depth: 20
            book_interval: 600
        trades: [BTC-USD, USDT-USD]
    OKEX:
        retries: 30
        l2_book:
            symbols: [BTC-USDT]
            max_depth: 20
            book_interval: 600
        trades: [BTC-USDT]
    HUOBI:
        retries: 30
        l2_book:
            symbols: [BTC-USDT]
            max_depth: 20
            book_interval: 600
        trades: [BTC-USDT]
    BINANCE:
        retries: 30
        l2_book:
            symbols: [BTC-USDT]
            max_depth: 20
            book_interval: 600
        trades: [BTC-USDT]

storage: [parquet]

parquet:
    del_file: true

    S3:
        endpoint: null
        key_id: A
        secret: m
        bucket: c
        prefix: null

storage_interval: 300
xiandong79 commented 4 years ago

https://github.com/go-redis/redis/issues/1131

That error is returned when Redis was restarted and loads data from the dump. Most likely your Redis is getting killed because it uses too much memory - probably you need to set maxmemory to limit memory usage.
bmoscon commented 4 years ago

@xiandong79 so there isnt an issue, right? it ran out of memory?