This patch started as a fix for overly ridged commit validation that rejected any index updates against commits from the past. In high volume ledgers, the indexer might not always keep up with commits so indexes might come back for commits > 2 t-values in the past. Rejecting these index updates had the effect of stopping the indexer until there's a pause in transactions, which had an impact on both query and transaction performance during periods of high load. This patch fixes that issue by adding a function that considers both the index and t values separately and takes the latest of both during each commit/index update.
In fixing this bug, I discovered another issue where the newly completed indexes were always thrown away and replaced with nil during each index index update because we stored commit and index state both in the branch and the branches latest-db record without ensuring that those two items stayed in sync. I have also fixed this issue, and I began to change our branch state to only store the current db without repeating any other state in branch metadata that needed to be kept in sync, but I ran into some circular dependency issues between connections, ledgers, branches, dbs, and indexers that started looking like a larger change, so I'm saving the rest of that for a later pr.
This patch started as a fix for overly ridged commit validation that rejected any index updates against commits from the past. In high volume ledgers, the indexer might not always keep up with commits so indexes might come back for commits > 2 t-values in the past. Rejecting these index updates had the effect of stopping the indexer until there's a pause in transactions, which had an impact on both query and transaction performance during periods of high load. This patch fixes that issue by adding a function that considers both the index and t values separately and takes the latest of both during each commit/index update.
In fixing this bug, I discovered another issue where the newly completed indexes were always thrown away and replaced with
nilduring each index index update because we stored commit and index state both in the branch and the branches latest-db record without ensuring that those two items stayed in sync. I have also fixed this issue, and I began to change our branch state to only store the current db without repeating any other state in branch metadata that needed to be kept in sync, but I ran into some circular dependency issues between connections, ledgers, branches, dbs, and indexers that started looking like a larger change, so I'm saving the rest of that for a later pr.