milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.4k stars 2.91k forks source link

[Bug]: Large growing segment #37387

Open bigsheeper opened 1 week ago

bigsheeper commented 1 week ago

Is there an existing issue for this?

Environment

- Milvus version: 2.4&2.5&master
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When inserts happen too quickly, segments may not have time to seal, resulting in the allocation of segments with maxRows (2.5GB). However, the ideal segment size should be maxRows * sealProportion. img_v3_02g4_4ab2a14f-bf08-497d-b060-d252ccb44d1g img_v3_02g4_e0188e62-2790-445a-8b37-6c2b23da049g

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

bigsheeper commented 4 days ago

/assign @yongpengli-z please help to verify