search

tardis-dev / serum-vial

Real-time WebSocket market data API for Serum
Mozilla Public License 2.0
173 stars 60 forks source link

Missing L3 Updates After Snapshot? #29

Closed chris-rr closed 3 years ago

chris-rr commented 3 years ago

Hi @thaaddeus ; firstly, thanks for the great work on this, awesome to have L3 streaming capability available for Serum ecosystem DEXs :)

Have been battling with an issue that seems to occur following some l3snapshot message publishing events; happens mostly after certain invalid l3diff detections, but has been seen also at initial subscription.

I am using the level2 stream in order to confirm the correctness of local reconstruction of the l3 orderbook (as kind of a "gap detection" mechanism in the absence of the message sequence numbers which would be used for CEX L3 streams); what I have observed is this:

  1. l3snapshot and l2snapshot are received and processed successfully; these agree perfectly (of course, as they are created at the same time from the same data)
  2. There is then in some cases an l2update message received, on which some price levels have been modified - however, no corresponding L3 updates (open/change/fill/done) are published at the same time. This obviously leads to a misalignment of the local L3 book with the published L2. I have defined my logic such that this triggers a forced reconnect; usually reconnecting to obtain a fresh snapshot and messages solves the issue, but have been instances where this repeated over several re-subscriptions.

Note this does not occur for every l3snapshot processed, and the agreement is perfect much of the time. Have unfortunately not managed to identify anything particularly special about the cases where it does occur - only thoughts I have had are that it may happen only when a particularly large update occurs directly after the snapshot is produced, or perhaps related to the time between the update producing the snapshot and the next change.

Any insight or advice you may be able to give on this would be much appreciated (even more so, a fix in the next version 😋)

thaaddeus commented 3 years ago

Hi @chris-rr can you share which version of serum-vial you're running? You're running it with your own RPC node?

chris-rr commented 3 years ago

@thaaddeus On the latest 1.2.5 version, not yet running our own node at this stage so using the public https://solana-api.projectserum.com

thaaddeus commented 3 years ago

Thanks, I'll try to investigate it, but don't have any ideas right now why it would be the case, perhaps will start with adding server side 'WARN' log for that scenario to see if it's happening on data mapping side or smth with publishing WS messages.

thaaddeus commented 3 years ago

That should be fixed in v1.2.8, thanks for the report.