algorandfoundation / algokit-subscriber-ts

Simple, but flexible / configurable Algorand transaction subscription / indexing mechanism
MIT License
12 stars 6 forks source link

Allow for controlled indexer catchup when processing large data volumes #32

Closed robdmoore closed 6 months ago

robdmoore commented 6 months ago

Add a maxIndexerRoundsToSync configuration that allows indexer catchup to be transactionally chunked into a given batch size of rounds. This allows for large pre-filters to be accommodated without memory and time blowing up and thus being caught in an infinite catchup loop. You may set that to, say 100,000 rounds.