milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications

https://milvus.io

Apache License 2.0

29.02k stars 2.79k forks source link

[Enhancement]: Concurrency control of clustering compaction #34544

Closed wayblink closed 1 month ago

wayblink commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

What would you like to be added?

34495

Clustering compaction is much bigger task than normal compaction. And milvus doesn't have a strict memory control. Hence, we need to control the concurrency of clustering compactions.

Why is this needed?

No response

Anything else?

No response

xiaofan-luan commented 1 month ago

what is the memory estimation of a clustering compaction? is it related to segment number * bin log size?

what is the expected memory usage and concurrency?