milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.36k stars 2.91k forks source link

[Bug]: Milvus standalone will crash when open all mmap on vector index and field, scalar index and field #36157

Closed NicoYuan1986 closed 5 days ago

NicoYuan1986 commented 2 months ago

Is there an existing issue for this?

Environment

- Milvus version: master-20240910-60f4db2f 
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Milvus standalone will crash when open all mmap on vector index and field, scalar index and field.

failed cases: https://qa-jenkins.milvus.io/blue/organizations/jenkins/existing-milvus/detail/existing-milvus/4/pipeline/

milvus pod: (on devops chaos-testing)

kubectl get pods |grep func-mmap-standalone-vlzph
func-mmap-standalone-vlzph-etcd-0                                 1/1     Running                           0                 4h16m
func-mmap-standalone-vlzph-milvus-standalone-7d468974fd-jj82q     0/1     CrashLoopBackOff                  45 (28s ago)      4h16m
func-mmap-standalone-vlzph-minio-9c5d55498-w82q7                  1/1     Running                           0                 4h16m

Expected Behavior

pass

Steps To Reproduce

No response

Milvus Log

https://grafana-devops.zilliz.cc/explore?orgId=1&left=%7B%22datasource%22:%22Loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bpod%3D%5C%22func-mmap-standalone-vlzph-milvus-standalone-7d468974fd-jj82q%5C%22%7D%22%7D%5D,%22range%22:%7B%22from%22:%22now-3h%22,%22to%22:%22now%22%7D%7D

Anything else?

No response

yanliang567 commented 2 months ago

/assign @sunby /unassign

NicoYuan1986 commented 1 month ago

I retest twice. not reproduce.

  1. master-20240919-e66beb7a-amd64 https://qa-jenkins.milvus.io/blue/organizations/jenkins/existing-milvus/detail/existing-milvus/5/pipeline
  2. master-20240918-167e4fb1 https://qa-jenkins.milvus.io/blue/organizations/jenkins/existing-milvus/detail/existing-milvus/6/pipeline

milvus standalone did not crash. But it still exists load timeout error.

xiaofan-luan commented 1 week ago

what is the progress for now?

NicoYuan1986 commented 5 days ago

not reproduced.