search

bmoscon / cryptofeed

Cryptocurrency Exchange Websocket Data Feed Handler
Other
2.22k stars 684 forks source link

Finer grained resets in response to sequence number misses on Coinbase #291

Closed zahnz closed 3 years ago

zahnz commented 4 years ago

Is your feature request related to a problem? Please describe. Currently, a reset and book snapshot is triggered for all pairs in a Coinbase feed if any of the pairs encounters a missing sequence number. Instead, what might be better is to only trigger the reset and subsequent REST book snapshot request for the pair(s) that had sequence number misses.

Describe the solution you'd like In coinbase.py, change __reset() and _book_snapshot() and the code that takes action on missing sequence numbers to work on a per-pair basis for missing sequence numbers.

Describe alternatives you've considered This could be coded around by adding one Coinbase feed per pair to the feedhandler, or splitting pairs up into many feeds (e.g. 3 pairs per feed). But, this has downsides: it will open some number of websocket connections to Coinbase per pair, which may be too resource intensive if many pairs need to be subscribed to. Nonetheless, it is a valid solution. However, I don't see any downsides to the suggested improvement above.

Additional context One potential consideration might be that sequence number misses are correlated in time, and upon missing one sequence number in one pair, others will be missed in other coins as well; therefore, resetting and requesting snapshots for all pairs in a feed would be the safe decision. This will need to be tested after any coding is done for this improvement.

I'm willing to do testing and coding on this.

bmoscon commented 4 years ago

its doable, its just a good amount of work to add this capability and then test it out.

bmoscon commented 4 years ago

to do this you'd need to implement a piece of new functionality:

unsubscribe - you'd need to unsubscribe from the channel/pair in question and then resubscribe.

Sequence would be:

  1. Detect missing data, clear/reset book (for just this pair)
  2. Unsubscribe from the channel/pair
  3. Ensure no pending data from this channel/pair (i.e. either wait for message back that unsub was successful or wait for something like half a second and ignore all messages for this pair)
  4. Resub to channel/pair like normal (including snapshot if necessary)
bmoscon commented 3 years ago

fixed (for coinbase). Other exchanges that support book snapshots via REST requests can probably benefit from the same treatment.