milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
31.04k stars 2.95k forks source link

[Bug]: After executing "rollout restart", the insert/search/query service will experience service degradation. #33876

Open zhuwenxing opened 5 months ago

zhuwenxing commented 5 months ago

Is there an existing issue for this?

Environment

- Milvus version:2.4-20240614-dff7ec7b-amd64
- Deployment mode(standalone or cluster):mixcoord
- MQ type(rocksmq, pulsar or kafka):kafka    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

image

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/rollout_restart_test/detail/rollout_restart_test/7/pipeline/78/

grafana link: https://grafana-4am.zilliz.cc/d/uLf5cJ3Ga/milvus2-0?orgId=1&var-datasource=P1809F7CD0C75ACF3&var-cluster=&var-namespace=chaos-testing&var-instance=kafka-mixcoord-7&var-collection=All&var-app_name=milvus&from=1718350677595&to=1718351967137

Anything else?

No response

xiaofan-luan commented 5 months ago

/assign @weiliu1031