milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.87k stars 2.94k forks source link

[Bug]: Mixcoord fails to start #37917

Open bigsheeper opened 1 day ago

bigsheeper commented 1 day ago

Is there an existing issue for this?

Environment

- Milvus version: 2.4
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The keep-alive fails, and there are slow logs in etcd. Maybe that an excessive amount of metadata is impacting etcd performance. dZanpeTlq4 TxiWMMQwLQ

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

xiaofan-luan commented 14 hours ago

how large is the etcd?

We may think of deploy etcd on seperate disk and there seems to be a perf issue on our test env

xiaofan-luan commented 14 hours ago

As far as I know, etcd is pretty stable on response for less than 10GB data