search

cowprotocol / dune-sync

Components for syncing off-chain data with Dune Community Sources
4 stars 1 forks source link

Delays when syncing data #72

Closed harisang closed 9 months ago

harisang commented 9 months ago

After PR #71, which fixed the issue with the backend database change, it seems to me that we might sync data with signficant delay, as the syncing, I believe, now depends on both prod and staging having at least one settlement every few hours. The reason for that, I think, is the min operator in this line:

https://github.com/cowprotocol/dune-sync/blob/d6f34e042f0426fac0f93d0d9a60c19c9d775df5/src/fetch/orderbook.py#L72

I would propose that we actually change it to max now. Let me know what you think @fhenneke

fhenneke commented 9 months ago

I would change it to max and remove the - REORG_THRESHOLD.

With the new query we look at blocks which have already successfully been processed (since these is an entry in settlement_observations). What can then go wrong is that maybe there are staging and production settlements on adjacent blocks and the later settlement is already processed but not the earlier one. Then the earlier one will not end up on dune. Not sure if that is actually happening sometimes. For that purpose we could have some delay, so something like - MAX_PROCESSING_DELAY with MAX_PROCESSING_DELAY = 1 or 2 or whatever seems fit. The old query had a delay of 65 which might included implicitly a MAX_PROCESSING_DELAY = 1 since the reorg threshold should be 64.