Open XuanYang-cn opened 2 months ago
Is there an existing issue for this?
- [x] I have searched the existing issues
Is your feature request related to a problem? Please describe.
Milvus can merge smaller segments into larger segment, but once a segment is flushed, it cannot be split into smaller segments.
This feature introducing a new compaction type: SplitCompaction. That segments can be splited into smaller one.
Auto trigger
- Input segments: size > maxSize * expansionRate
- Ouput segments: size ~= maxSize
Trigger
- timely
- manually
Describe the solution you'd like.
A compaction has
n
input segments, hasm
output segments.
- Previous compactions:
n >= 1, m == 1
- This feature introduces:
n == 1, m >= 1
- Eventually:
n >= 1, m >= 1
Describe an alternate solution.
No response
Anything else? (Additional Context)
No response
I don't think we need a special compaction type? for any of the compaction, if output is > segment size we should split it automatically
@xiaofan-luan Not designed to be a new type, just calling it split-compaction for a brand new trigger and avoid changing mix_compactor
to influence all MixCompactions.
This's a decision compromised for:
If given enough time, not aiming at 2.4.x, I'll just change it to the eventually goal. Making it just like an enhanced MixCompaction.
This issue description is unclear, add some notes here:
@XuanYang-cn
Did we test how much memory it will cost to compact on large segment like 100G?
Is there an existing issue for this?
Is your feature request related to a problem? Please describe.
Milvus can merge smaller segments into larger segment, but once a segment is flushed, it cannot be split into smaller segments.
This feature introducing a new compaction type: SplitCompaction. That segments can be splited into smaller one.
Auto trigger
Trigger
Describe the solution you'd like.
A compaction has
n
input segments, hasm
output segments.n >= 1, m == 1
n == 1, m >= 1
n >= 1, m >= 1
Describe an alternate solution.
No response
Anything else? (Additional Context)
No response