Today ApplyContext has prepare_for and finish_for, two APIs that are called for the start and end of each region's apply work. We can do the check in the prepare_for function with region's range. The tricky part is how to carry the should_write_to_memory_engine boolean flag and let the WriteBatch be aware of it. We may have to introduce something like cache_enable() method into WriteBatch trait. And for each region's start, this cache_enable will be reset accordingly.
This could be an Option<bool>: when there's no context, we just use the current way that filters the keys one by one.
This can be done later after the performance comparison.
Tasks
[ ] Spike: basic performance tests for write path, initial implementation of this optimization for feasibility test.
[ ] Production implementation:
[ ] Change WriteBatch to have a cache_enabled() -> Option<bool> and enable_cache_check(start_key: &[u8], end_key: &[u8]), implemented in Memory Engine as described above. No-op for regular engine.
[ ] Change ApplyContext to update WriteBatch internal state in ApplyContext's prepare_for and finish_for
afeinberg
commented
9 months ago
Do not filter every single key in buffer when writing
Parent issue: https://github.com/tikv/tikv/issues/16141
Overview
Today
ApplyContexthasprepare_forandfinish_for, two APIs that are called for the start and end of each region's apply work. We can do the check in theprepare_for functionwith region's range. The tricky part is how to carry theshould_write_to_memory_engineboolean flag and let theWriteBatchbe aware of it. We may have to introduce something likecache_enable()method intoWriteBatchtrait. And for each region's start, thiscache_enablewill be reset accordingly.This could be an
Option<bool>: when there's no context, we just use the current way that filters the keys one by one. This can be done later after the performance comparison.Tasks
[ ] Spike: basic performance tests for write path, initial implementation of this optimization for feasibility test.
[ ] Production implementation:
cache_enabled() -> Option<bool>andenable_cache_check(start_key: &[u8], end_key: &[u8]), implemented in Memory Engine as described above. No-op for regular engine.ApplyContext'sprepare_forandfinish_for