It was noticed that snapshot request on Binance could be lost when the exchange's rate-limiter had been triggered. This would lead to memory leak due to the MbP Sequencer collecting incremental updates while waiting for the snapshot. The was no back-stop implemented to avoid collecting "forever".
This is the new MbP Sequencer logic
When incremental updates are added first time ==> request for snapshot
When snapshot received ==> correlate on sequence number and apply all incremental updates collected after
When incremental received and no snapshot was received before timeout ==> reset + request for snapshot
For the roq-binance gateway one can use --mbp_sequencer_timeout flag to control this. Default is 120 seconds.
There is a also an option to cap the number of updates collected in memory, but it's fairly high. Timeout should be the preferred option to control this.
It was noticed that snapshot request on Binance could be lost when the exchange's rate-limiter had been triggered. This would lead to memory leak due to the MbP Sequencer collecting incremental updates while waiting for the snapshot. The was no back-stop implemented to avoid collecting "forever".
This is the new MbP Sequencer logic