search

hirosystems / chainhook

Extract transactions from Stacks and Bitcoin and build event driven re-org resistant indexers and databases.
GNU General Public License v3.0
155 stars 63 forks source link

Use state to avoid re-sending data to observers #554

Open lgalabru opened 7 months ago

lgalabru commented 7 months ago

When a chainhook-node is re-processing blocks (for whatever reason), it should avoid re-sending blocks to observers when they have been sent in the past. Data from Redis could help.

andresgalante commented 7 months ago

We might need to wait after the end of April (BTC++ conf) to take on this one

MicaiahReid commented 4 months ago

To be honest, I don't know exactly how this one should be approached. But here are some notes:

We already have this to some degree when a Chainhook node is restarted. When a predicate is in Streaming mode, we update the status of the predicate after every block, storing the last evaluated block. When a predicate is in Scanning mode, we update the status after every match between a block + predicate. With these values stored, whenever Chainhook is restarted, the service can load the predicates from redis, evaluate each predicate's status to determine the last block evaluated, and continue where it left off. This prevents a restart of the Chainhook node from re-sending data to observers.

This issue, however, is for another case (which happened recently). If a Stacks node has to be reset, say to chain tip - 100 blocks, each of those 100 blocks will have already been evaluated by Chainhook. However, when the block is received by the event observer there currently is no way to determine if the predicate has already been evaluated against that block (the Chainhook SDK does not have access to the redis store of predicate status). Furthermore, even if the predicate has a last_evaluated_block, in many cases (such as a reorg), we should still evaluate the block.

It would be nice to have some guarantee that a predicate will never be triggered twice from the same block, but I think it will be quite difficult and is currently handled for the general case. The recent events on the testnet that caused us to reset the chain will probably be very rare, so maybe we aren't as concerned about handling it.

I wouldn't say I recommend closing this issue at this point, but maybe it can be revisited in a few months