search

gmanroney / cryptoboard

Collects data from multiple cryptocurrency exchanges, transforms it and posts it into redis
MIT License
11 stars 2 forks source link

Intermittent bug with BITFINEX exchange feed #12

Open gmanroney opened 6 years ago

gmanroney commented 6 years ago

Basic function working but it crashes some time after start with the following error:

BITFINEX wss://api.bitfinex.com/ws BTCUSD BITFINEX wss://api.bitfinex.com/ws LTCUSD BITFINEX wss://api.bitfinex.com/ws LTCBTC BITFINEX wss://api.bitfinex.com/ws ETHUSD BITFINEX wss://api.bitfinex.com/ws ETHBTC BITFINEX wss://api.bitfinex.com/ws ETCUSD BITFINEX wss://api.bitfinex.com/ws ETCBTC BITFINEX wss://api.bitfinex.com/ws BFXUSD BITFINEX wss://api.bitfinex.com/ws BFXBTC BITFINEX wss://api.bitfinex.com/ws RRTUSD BITFINEX wss://api.bitfinex.com/ws RRTBTC BITFINEX wss://api.bitfinex.com/ws ZECUSD BITFINEX wss://api.bitfinex.com/ws ZECBTC Finshed launch Connected to Redis Server events.js:160 throw er; // Unhandled 'error' event ^ Error: unexpected server response (429) at ClientRequest._req.on (/Users/germoroney/Development/cryptoboard/node_modules/ws/lib/WebSocket.js:653:21) at emitOne (events.js:96:13) at ClientRequest.emit (events.js:188:7) at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:473:21) at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23) at TLSSocket.socketOnData (_http_client.js:362:20) at emitOne (events.js:96:13) at TLSSocket.emit (events.js:188:7) at readableAddChunk (_stream_readable.js:176:18) at TLSSocket.Readable.push (_stream_readable.js:134:10) [nodemon] app crashed - waiting for file changes before starting...

The number of clients connecting to redis to read data were initially thought to be related but this was found to be misleading. Happens with 1, 2 or 3 clients connecting to REDIS.

Error message found in https://github.com/bitfinexcom/bitfinex-api-node seems to be related to BITFINEX service itself ....

gmanroney commented 6 years ago

Discovered that published messages on redis are never saved and if no subscriber are lost forever. So doing an expire is not relevant and its not going to take up space on a redis server https://stackoverflow.com/questions/44134104/can-to-set-expire-time-for-a-redis-pub-sub-message

There is a problem somewhere with connections and / or loss of internet connection but yet to determine root cause

There is also an issue with when it fails you need to stop and restart redis. May be something to do with the redis infrastructure setup to investigate too .