Closed zyy17 closed 2 weeks ago
I make some stupid(git reflog
save me again) git operations and suddenly can't reopen the original PR #4124. I had no choice but to create a new PR. I'm sorry for the confusion.
[!NOTE]
Reviews paused
Use the following commands to manage reviews:
@coderabbitai resume
to resume automatic reviews.@coderabbitai review
to trigger a single review.
The recent changes introduce asynchronous functionality and plugins to the compaction feature of the Mito engine. They include new structures for remote job scheduling and result handling, with alterations to various modules to support these features. The modifications enhance the flexibility of the compaction process, allowing for both local and remote task handling, and improve error management within the system.
Files/Modules | Change Summary |
---|---|
src/mito2/src/compaction.rs |
Added Plugins , made some functions async, and handled remote compaction fallback to local compaction. |
src/mito2/src/engine.rs |
Added plugins parameter to MitoEngine and EngineInner constructors. |
src/mito2/src/error.rs |
Added RemoteJobScheduler , OutputTx , and ParseJobId to Error enum, and handled these variants in error extension implementation. |
src/mito2/src/region/options.rs |
Added remote_compaction function and remote_compaction field to TwcsOptions struct. |
src/datanode/src/datanode.rs , src/mito2/src/test_util.rs |
Modified functions to include Plugins parameters where applicable. |
src/mito2/src/compaction/compactor.rs |
Updated open_compaction_region function to use ObjectStoreManagerRef . |
src/store-api/src/mito_engine_options.rs |
Added "compaction.twcs.remote_compaction" to is_mito_engine_option_key function check list. |
src/mito2/src/schedule/remote_job_scheduler.rs |
Added remote job scheduling functionality, defining traits, job types, and notification mechanisms. |
sequenceDiagram
participant PluginManager
participant CompactionScheduler
participant RemoteJobScheduler
participant Worker
participant ErrorHandler
PluginManager->>CompactionScheduler: Initialize with Plugins
CompactionScheduler->>RemoteJobScheduler: Schedule Compaction (async)
RemoteJobScheduler->>Worker: Execute Job
Worker-->>RemoteJobScheduler: Job Result (async)
RemoteJobScheduler->>CompactionScheduler: Job Finished Notification (async)
CompactionScheduler->>ErrorHandler: Handle Errors (if any)
🐇 In lines of code, a tale unfolds, Of compaction tasks both brave and bold. With async calls and plugins near, Remote jobs now can interfere. Errors handled, smooth and bright, Our Mito engine takes new flight. 🚀
[!TIP]
Early access features: enabled
We are currently testing the following features in early access: - **OpenAI `gpt-4o` model for code reviews and chat**: OpenAI claims that this model is better at understanding and generating code than the previous models. We seek your feedback over the next few weeks before making it generally available. Note: - You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file. - Please join our [Discord Community](https://discord.com/invite/GsXnASn26c) to provide feedback and report issues. - OSS projects are currently opted into early access features by default.
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?
Attention: Patch coverage is 59.27419%
with 101 lines
in your changes missing coverage. Please review.
Project coverage is 84.52%. Comparing base (
e531326
) to head (0f9335d
).
Mostly LGTM. @fengjiachun @v0y4g3r @evenyag PTAL and add comments from your perspective
@coderabbitai pause
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
RemoteJobScheduler
For the storage disaggregated system, we can always offload the CPU-intensive and IO-intensive tasks(for example, compaction and index) to the remote workers. For the above scenario, the PR introduces the abstraction.
RemoteJobScheduler
is a trait that defines the API for scheduling remote jobs. Its implementation is in GreptimeDB Enterprise.The PR modify
schedule_compaction_request()
to support remote compaction:remote_compaction
inregion_options
and theRemoteJobScheduler
is initialized, the Mito will execute remote compaction;Other changes
Add the
async
keyword for all the compaction-related functions because theschedule_compaction_request
needs to beasync
;Use
Option
type forsenders
inCompactionFinished
because we don't need it in remote compaction scenario;Add
remote_compaction
in compaction options;TODOs
RemoteJobScheduler
from the plugin system;Design the API to fetch the Jobs from the scheduler. When the datanode restarts, it can rebuild the context of the remote job;Add the unit tests for the;RemoteJobScheduler
Checklist
Summary by CodeRabbit
New Features
Plugins
functionality to enhance compaction scheduling and handling.Enhancements
MitoEngine
andEngineInner
constructors to acceptplugins
parameters.RemoteJobScheduler
and other error variants.remote_compaction
in compaction options.Bug Fixes