Tracking issue for compaction offloading

LeslieKid commented 3 months ago

Describe This Problem

We found in production that the speed of sst compaction is unable to keep up with the speed of sst generation, leading to poor query performance. However we are unable give more resource to compaction to solve the problem because query/write is more important than compaction in the same node.

It is really hard to do a trade-off about resource allocation among query, write and compaction in lsm model. We want to compact the generated small ssts as fast as possible, but we can't tolerate its influence to query/write. And finally I think offload the compaction to the seperated nodes may be the key for it.

Proposal

The following is the architecture for compaction offloading.

To support compaction offloading, we need:

[ ] Horaedb node supports submitting the compaction task to remote #1563
- Introduce runnable compaction node and expose the api for horaedb node to ask for remote compaction service (Execute Plane)
- Impl remote mode compactor supporting compaction offload
[ ] Horaemeta supports managing the compaction nodes (Control Plane) #1559
- Impl the ability to monitor the compaction servers (Monitor)
- Impl scheduling algorithm for load balance and expose the api for horaedb node to get the proper remote compaction node (Scheduler)
[ ] Integration tests for compaction #1573
- See #1571 for more details

Additional Context

This issue replaces issue #1480. Please close issue #1480 as it is outdated.

incubator-horaedb-proto#133 is highly related to this issue.

Rachelint commented 3 months ago

looks great!

Rachelint commented 2 months ago

I think in the first stage, we can treat the compaction cluster as same as the horaedb cluster. And we just reuse the ability about monitoring and scheduling for horaedb cluster in horaemeta.

And we can start to desigin the specific schedule strategy for compaction cluster after it can work.

LeslieKid commented 2 months ago

I think in the first stage, we can treat the compaction cluster as same as the horaedb cluster. And we just reuse the ability about monitoring and scheduling for horaedb cluster in horaemeta.

And we can start to desigin the specific schedule strategy for compaction cluster after it can work.

Thanks for your suggestion! I think it's a good idea to first implement a simple working version.

Rachelint commented 1 month ago

I think Horaedb node supports submitting the compaction task to remote may be similar as Compaction node supporting remote compaction service.

I guess the third step can be integration test, test is very important actually.

Rachelint commented 1 month ago

Maybe you will be insterested when writing test https://github.com/apache/horaedb/tree/main/integration_tests/dist_query

apache / horaedb