Closed v0y4g3r closed 1 week ago
[!IMPORTANT]
Review skipped
Review was skipped due to path filters
Files ignored due to path filters (1)
* `Cargo.lock` is excluded by `!**/*.lock`You can disable this status message by setting the
reviews.review_status
tofalse
in the CodeRabbit configuration file.
The updates include the addition and enhancement of methods to the InvertedIndexReader
trait, restructuring the CacheManager
to manage index caching, introducing new modules for index caching, and modifying test cases accordingly. Additionally, it includes changing various signature methods and utilizing Arc
to manage shared metadata efficiently.
File/Path | Change Summary |
---|---|
src/index/Cargo.toml |
Added uuid as a workspace dependency. |
src/index/src/inverted_index/.../reader.rs |
Added read_all and seek_read methods and modified return types for metadata , fst , and bitmap . |
src/index/src/inverted_index/.../blob.rs |
Adjusted method signatures to use Arc for metadata and renamed methods for reading data. |
src/index/src/inverted_index/.../predicates_apply.rs |
Modified mock_metas function to use Arc for shared metadata ownership. |
src/mito2/src/cache.rs |
Added index module with caching declarations and methods in CacheManager and CacheManagerBuilder . |
src/mito2/src/cache/index.rs |
Introduced CachedInvertedIndexBlobReader struct and related caching implementations. |
src/mito2/src/config.rs |
Added cache size fields for inverted index metadata and content to MitoConfig . |
src/mito2/src/read/scan_region.rs |
Introduced index_cache variable and passed it to SstIndexApplierBuilder . |
src/mito2/src/sst/file.rs |
Added conversion implementation from FileId to Uuid . |
src/mito2/src/sst/.../applier.rs |
Added inverted_index_cache field, modified apply method, and updated blob reader creation conditions. |
src/mito2/src/sst/.../builder.rs |
Added index_cache field and parameter to SstIndexApplierBuilder . |
src/mito2/src/sst/.../between.rs |
Updated test cases to include None for the index_cache parameter. |
src/mito2/src/sst/.../comparison.rs |
Updated test cases to include None for the index_cache parameter. |
src/mito2/src/sst/.../eq_list.rs |
Updated test cases to include None for the index_cache parameter. |
src/mito2/src/sst/.../in_list.rs |
Updated test cases to include None for the index_cache parameter. |
src/mito2/src/sst/.../regex_match.rs |
Updated test cases to include None for the index_cache parameter. |
src/mito2/src/sst/.../creator.rs |
Added usage of InvertedIndexCache and passed it to the builder. |
src/mito2/src/worker.rs |
Added index metadata and content size configurations. |
tests-integration/tests/http.rs |
Added configuration settings for cache sizes for inverted index metadata and content. |
In the lines of Rust we weave, With bytes and cache we achieve, Indexes cached, with data vast, Efficiency and speed to outlast.
Oh, code of magic, so refined, Full of treasures left behind, We celebrate this feat so grand, In the realm of data we now stand. 🌟
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?
Do we have any benchmark program and results for this PR? @v0y4g3r
Do we have any benchmark program and results for this PR? @v0y4g3r
I updated the PR description. This PR improves those queries which are already fast, but not fast enough, namely those 5~6ms queries. An improvement of 2 to 3ms means cutting the total cost time by half.
Attention: Patch coverage is 91.07981%
with 19 lines
in your changes missing coverage. Please review.
Project coverage is 84.85%. Comparing base (
e5730a3
) to head (b42a932
). Report is 1 commits behind head on main.
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
This PR adds im-memory cache for inverted index that boost most queries by reducing the pruning cost, which reduces total cost by 2~3ms:
Future work
PuffinReader
has been refactored recently and I need to find a proper way to passInvertedIndexCache
toBoundedStager
.Checklist
Summary by CodeRabbit
New Features
Improvements
SstIndexApplier
to conditionally use the cache for index reading.Arc
.Configuration Updates
Testing Enhancements