coderabbitai[bot]
commented
1 week ago [!NOTE]
Reviews paused
Use the following commands to manage reviews:
@coderabbitai resumeto resume automatic reviews.@coderabbitai reviewto trigger a single review.
Walkthrough
The changes introduce and integrate comprehensive full-text indexing capabilities into the Mito engine. Configurations for index creation during flush, compaction, and querying, along with memory thresholds, are introduced. New public constants and methods handle the configuration schema and error-handling mechanisms. The full-text index functionalities are integrated across key components, including compaction, flushing, and caching processes, enhancing the overall indexing and search capabilities of the engine.
Changes
| Files | Change Summary |
|---|---|
config/config.md, config/datanode.example.toml, config/standalone.example.toml |
Introduced configuration options for full-text indexing, including settings for index creation on flush, compaction, and query, and memory threshold for index creation. |
src/datatypes/src/schema.rs, src/datatypes/src/schema/column_schema.rs |
Added FulltextAnalyzer and FulltextOptions, constants for full-text keys, and methods in ColumnSchema for retrieving full-text options. |
src/mito2/src/access_layer.rs, src/mito2/src/cache/write_cache.rs |
Incorporated fulltext_index_config into the AccessLayer and WriteCache structs. |
src/mito2/src/compaction/compactor.rs, src/mito2/src/flush.rs |
Added handling of fulltext_index_config in compaction and flush tasks, integrating full-text indexing into these processes. |
src/mito2/src/config.rs, src/mito2/src/error.rs |
Added FulltextIndexConfig struct with configurations and new error variants for full-text indexing. |
src/mito2/src/sst/file.rs, src/mito2/src/sst/index.rs, src/mito2/src/sst/index/fulltext_index.rs |
Included FulltextIndex in FileMeta, IndexType, and added methods for full-text index availability and creation. |
src/mito2/src/sst/index/fulltext_index/creator.rs |
Introduced SstIndexCreator and SingleCreator structs for creating and managing full-text indexes, with methods for updating, finishing, and aborting index creation. |
src/mito2/src/sst/index/indexer/abort.rs, src/mito2/src/sst/index/indexer/finish.rs, src/mito2/src/sst/index/indexer/update.rs |
Added functions for aborting, finishing, and updating full-text indexes within the Indexer implementation. |
src/mito2/src/sst/index/intermediate.rs, src/mito2/src/test_util.rs |
Added base_dir to IntermediateManager and methods for generating fulltext index paths. Adjusted test utilities for new full-text configurations. |
tests-integration/tests/http.rs |
Added configuration section [region_engine.mito.fulltext_index] with settings for full-text indexing in integration tests. |
Sequence Diagram(s)
sequenceDiagram
participant User
participant Configuration
participant MitoEngine
participant Indexer
participant Storage
User->>Configuration: Configure full-text index settings
Configuration->>MitoEngine: Pass settings
MitoEngine->>Indexer: Initialize with full-text index config
Indexer->>Storage: Create/Update indexes during flush or compaction
Storage-->>Indexer: Return index metadata
Indexer-->>MitoEngine: Update active index
MitoEngine-->>User: Provide search capabilitiesPoem
In the engine, new indexes bloom, Full-text now pierces the query's gloom. On flush and compaction, they take their place, Enhancing search with elegant grace. Memory thresholds guide the way, For faster finds, without delay. Here's to data, swift and bright, 🐰 A joyful hop to indexing's new height! 📚✨
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?
zhongzc
codecov[bot]
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
4246
What's changed and what's your intention?
Checklist
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Tests
Chores