WIPish but parking this here for now. This is a very quick and dumb performance enhancement for header validation, which implements a simple static cache in the Difficulty iterator, which caches block headers read from the database as they're read one-by-one to avoid the need to read them again on the next set of difficulty calculations when validating headers sequentially. There is no logic to how headers are cached, and there's a simple limit of 10k headers for the time being. When the limit is hit the iterator will simple read from the DB again for the next iterator.
Advantage of this is that it's simple and gives an additional 5 second speed-up from the results in #3671, bringing the 100k test to about 19 seconds on the test hardware. Further performance enhancements will likely require more careful changes to consensus code.
Flamegraph without this cache (all enhancements to #3671 merged): about 24 seconds for 100k test:
And with this change (about 19 seconds), note the time spent on mdb_lock is now reduced to a sliver:
WIPish but parking this here for now. This is a very quick and dumb performance enhancement for header validation, which implements a simple static cache in the Difficulty iterator, which caches block headers read from the database as they're read one-by-one to avoid the need to read them again on the next set of difficulty calculations when validating headers sequentially. There is no logic to how headers are cached, and there's a simple limit of 10k headers for the time being. When the limit is hit the iterator will simple read from the DB again for the next iterator.
Advantage of this is that it's simple and gives an additional 5 second speed-up from the results in #3671, bringing the 100k test to about 19 seconds on the test hardware. Further performance enhancements will likely require more careful changes to consensus code.
Flamegraph without this cache (all enhancements to #3671 merged): about 24 seconds for 100k test:
And with this change (about 19 seconds), note the time spent on mdb_lock is now reduced to a sliver: