search

hirosystems / chainhook

Extract transactions from Stacks and Bitcoin and build event driven re-org resistant indexers and databases.
GNU General Public License v3.0
143 stars 57 forks source link

Investigate why the Chainhook chain segment can become outdated #623

Open csgui opened 2 months ago

csgui commented 2 months ago

An issue has happened, in production, where the Chainhook’s chain segment became outdated.

And since the code base is not prepared to handle that, only restoring the Chainhook's chain state solves this issue.

https://github.com/hirosystems/chainhook/blob/main/components/chainhook-sdk/src/indexer/mod.rs#L255 https://github.com/hirosystems/chainhook/blob/main/components/chainhook-sdk/src/indexer/mod.rs#L414

The following log messages where triggered in the case of that situation.

Will have to fork: OutdatedSegment
Unable to process Stacks Block #158000 (0x089f...445b) - inboxed for later
unable to infer chain progress
...
Will have to fork: OutdatedSegment
Unable to process Stacks Block #157999 (0x2ee0...087b) - inboxed for later
unable to infer chain progress
...
Will have to fork: OutdatedSegment
Unable to process Stacks Block #157998 (0x4353...70af) - inboxed for later
unable to infer chain progress

The work here is to find the root cause for why this can happen.

smcclellan commented 1 month ago

I see two issues here:

  1. A spike to investigate why the chain state became outdated, ideally reproducing the issue
  2. Make the system more resilient so it can recover from an outdated chain state

Let's have this ticket be the first one, and I'll create a separate ticket for the second one.