When top-down finality lags behind for whatever reason, it'll get increasingly slower to fetch blocks when we recover, since those portions of the parent chain will be deeper and deeper into history, something that Lotus tends to struggle with.
Fetching a block height involves fetching two kinds of logs: validator changes and top down messages. We already fetch both in parallel. However, there's significant room for optimization.
See logs below from a deployment that took 6 minutes to fetch 22 Filecoin mainnet blocks into the top-down cache.
Proposals
When doing deep catch-ups, we can easily amortize the RPC query costs by calling eth_getLogs for height ranges (e.g. 10, 20, 30 blocks at a time, configurable) instead of individual blocks.
We can diff the top down nonce and configuration number between the start and end of the next range, to potentially even skip the range if nothing has changed.
Context
When top-down finality lags behind for whatever reason, it'll get increasingly slower to fetch blocks when we recover, since those portions of the parent chain will be deeper and deeper into history, something that Lotus tends to struggle with.
Fetching a block height involves fetching two kinds of logs: validator changes and top down messages. We already fetch both in parallel. However, there's significant room for optimization.
Proposals
eth_getLogsfor height ranges (e.g. 10, 20, 30 blocks at a time, configurable) instead of individual blocks.Logs