milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
28.1k stars 2.71k forks source link

[Bug]: [benchmark][standalone] build IVF_FLAT index failed ` rpc error: code = DeadlineExceeded desc = context deadline exceeded` #32400

Open wangting0128 opened 2 months ago

wangting0128 commented 2 months ago

Is there an existing issue for this?

Environment

- Milvus version: test_hybrid_search_serial_ivf_flat_hnsw_standalone
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq   
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: multi-vector-based-scene1-f8pw5 test case name: test_hybrid_search_serial_ivf_flat_hnsw_standalone

server: init stats

NAME                                                              READY   STATUS                            RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-based-scene1-f8pw5-etcd-0                            1/1     Running                           0               2m33s   10.104.15.16    4am-node20   <none>           <none>
multi-vector-based-scene1-f8pw5-milvus-standalone-65cf6f86z2s9q   1/1     Running                           0               2m33s   10.104.26.10    4am-node32   <none>           <none>
multi-vector-based-scene1-f8pw5-minio-6f9756b97c-mfzmr            1/1     Running                           0               2m33s   10.104.15.19    4am-node20   <none>           <none>

after testing

NAME                                                              READY   STATUS                            RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-based-scene1-f8pw5-etcd-0                            1/1     Running                           0               24h     10.104.15.16    4am-node20   <none>           <none>
multi-vector-based-scene1-f8pw5-milvus-standalone-65cf6f86z2s9q   1/1     Running                           3 (17h ago)     24h     10.104.26.10    4am-node32   <none>           <none>
multi-vector-based-scene1-f8pw5-minio-6f9756b97c-mfzmr            1/1     Running                           0               24h     10.104.15.19    4am-node20   <none>           <none>
截屏2024-04-17 11 29 49

client pod name: multi-vector-based-scene1-f8pw5-19328334 client log:

截屏2024-04-17 11 24 41

Expected Behavior

No response

Steps To Reproduce

1. create a collection, 8 fields: "id", "float_vector", "float_vector_1", "int64_1", "int64_2", "float_1", "double_1", "varchar_1"
2. build index
   IVF_FLAT: float_vector
   HNSW: float_vector_1
   INVERTED: "int64_1", "int64_2", "float_1", "double_1", "varchar_1"
3. insert 25m data
4. flush collection
5. build index again with the same params <- failed

Milvus Log

No response

Anything else?

server config:

截屏2024-04-17 11 29 28
xiaofan-luan commented 2 months ago

/assign @xiaocai2333

xiaofan-luan commented 2 months ago

/assign @xiaocai2333

yanliang567 commented 2 months ago

/unassign

xiaocai2333 commented 1 month ago

It seems that RootCoord crashed due to some other reasons.