search

filecoin-project / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
2.82k stars 1.25k forks source link

Enable automatic backfilling of sqlite databases (msgindex, events, txhash) #11007

Open fridrik01 opened 1 year ago

fridrik01 commented 1 year ago

Due to the importance of having these indices up-to-date from a performance perspective (API providers), we should make sure that we always have these databases fully backfilled.

Currently we don't automatically backfill the msgindex/events/txhas sqlite databases but instead rely on the user knowing to backfill manually. Having these databases fully backfilled is important to the performance of several RPC methods which otherwise have a huge latency (over 10+ minutes on archival nodes).

Instead, on Lotus startup, we should just always check if we have missing data (handling also cases where config was changed from disabled to enabled for an index) and then handle the backfilling in the background automatically. Only in some rare edge cases should user need to manually run backfill via lotus-shed, for example if they are migrating dbs and/or editing the raw sqlite data.

rvagg commented 1 month ago

This probably needs something like MaxBackfillLookback for the case of an archival node turning this on late in the game; maybe you don't want to go back to FEVM and reprocess all tipsets to get events that you don't have.