milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.87k stars 2.94k forks source link

[Bug]: GetIndexInfos RPC is called too frequently #37634

Open bigsheeper opened 1 week ago

bigsheeper commented 1 week ago

Is there an existing issue for this?

Environment

- Milvus version: 2.4&master
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The IndexChecker in querycoord calls the GetIndexInfos RPC too frequently. image

Expected Behavior

We should batch segments into a single GetIndexInfos RPC call to avoid frequent RPCs. On the other hand, the frequency of the index checker (currently every 10 seconds) may could be reduced. https://github.com/milvus-io/milvus/blob/30d396f476598a204c4e77ad4de85495ce51d241/internal/querycoordv2/checkers/index_checker.go#L135-L141

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

bigsheeper commented 1 week ago

related to: https://github.com/milvus-io/milvus/issues/37630

xiaofan-luan commented 1 week ago

@bigsheeper

sems that we can remove the broker since datacoord and indexcoord are now merged?

bigsheeper commented 1 week ago

@bigsheeper

sems that we can remove the broker since datacoord and indexcoord are now merged?

@xiaofan-luan These GetIndexInfos invokes are from querycoord.