[Bug]: [major] The datanode is continously restarted for OOM after major compaction for 50M (768) dataset

[binbinlv]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: 2.4-latest
- Deployment mode(standalone or cluster):both
- MQ type(rocksmq, pulsar or kafka):    all
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus-latest
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The datanode is continously restarted for OOM after major compaction for 50M (768) dataset

Expected Behavior

No crash

Steps To Reproduce

1. create collection with partition key enabled (partition key as clustering key)
2. insert 50M (768)
3. major compaction

Milvus Log

No response

Anything else?

No response

[xiaocai2333]

The failure to reset the flushedRowNum of the writerBuffer resulted in subsequent segments in this buffer continually meeting the pack conditions, leading to the creation of numerous segments. The datanode couldn't load the BloomFilters for so many segments, causing an OOM error.

[binbinlv]

The OOM issue still exists when major compaction on 50M (768) data: image: v2.4.6-nightly-176034d1-872