milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.52k stars 2.83k forks source link

[Bug]: bulksearch with (nq=500,nlist=1024,nprobe=32) is very slow,feature is orb feature with 256 dim and indexed by BIN_IVF_FLAT #34997

Closed kkpssr closed 1 month ago

kkpssr commented 1 month ago

Is there an existing issue for this?

Environment

- Milvus version:2.4
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: AMD 5950,128G 
- GPU: 
- Others:

Current Behavior

the performance is very instability, sometime cost 800ms or more, sometimes cost 300ms in 2m data ,but is far from example in FAQ shows only cost 200ms when nq=1000 in 50m data.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

kkpssr commented 1 month ago

repeat

xiaofan-luan commented 1 month ago

There are might be many reasons

  1. too many segments, you can change segment size to 2G see if it works.
  2. cpu doesn't support SIMD -> I think 5950 should at least support AVX2, but might not AVX512

if you can offer pprof or minitoring that could be really helpful