The loader needs to be more aware of the chain state. Currently it won't handle loading at the head of the chain well.
This can be fixed by modifying the Exfiltrator such that when the end_at is None that it detects when it reaches the HEAD of the chain and switches to a more intelligent polling mechanism to wait for the next block.
Intelligent polling is probably best done by measuring/tracking the block time for recent blocks and polling based on something like the 99th percentile for block times.
The
loaderneeds to be more aware of the chain state. Currently it won't handle loading at the head of the chain well.This can be fixed by modifying the
Exfiltratorsuch that when theend_at is Nonethat it detects when it reaches the HEAD of the chain and switches to a more intelligent polling mechanism to wait for the next block.Intelligent polling is probably best done by measuring/tracking the block time for recent blocks and polling based on something like the 99th percentile for block times.