search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
31.04k stars 2.95k forks source link

[Bug]: Insert is slow #37633

Open bigsheeper opened 2 weeks ago

bigsheeper commented 2 weeks ago

Is there an existing issue for this?

Environment

- Milvus version: master&2.4
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When there are many collections and partitions, the AssignSegment operation becomes very slow, which in turn causes insertion to slow down significantly. image image image image

Expected Behavior

The AllocSegment function in SegmentManager is slow, likely due to lock contention. app1Lw4ZGb

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

bigsheeper commented 2 weeks ago

related to: https://github.com/milvus-io/milvus/issues/37630

bigsheeper commented 2 weeks ago

We should reduce the lock granularity in SegmentManager.