[Bug]: [Perf][cluster]Milvus HNSW's QPS for search in distributed decreases gradually

[jingkl]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

deploy_config： fouramf-server-cluster-8c16m case_params： fouramf-client-sift1m-concurrent-hnsw other_params： --milvus_tag_prefix=2.2.0 -s --deploy_mode=cluster case_name： test_concurrent_locust_custom_parameters

This is the result of a week of hnsw performance testing, in this week, hnsw performance QPS gradually decreased, latency gradually became higher

Expected Behavior

No response

Steps To Reproduce

1. create a collection 
        2. build hsnw index on vector column
        3. insert 1000000 vectors
        4. flush collection
        5. build index on vector column with the same parameters
        6. build index on on scalars column or not
        7. count the total number of rows
        8. load collection
        9. Concurrent [1, 20, 50, 100] for search operation
        10. clean all collections

Milvus Log

No response

Anything else?

[2023-03-18 00:37:40,540 - INFO - fouram]: [check_params] scene_concurrent_locust required params: {'dataset_params': {'dim': 128, 'dataset_name': 'sift', 'dataset_size': 1000000, 'ni_per': 50000, 'metric_type': 'L2'}, 'collection_params': {'other_fields': []}, 'load_params': {}, 'search_params': {}, 'index_params': {'index_type': 'HNSW', 'index_param': {'M': 8, 'efConstruction': 200}}, 'concurrent_params': {'concurrent_number': [1, 20, 50, 100], 'during_time': 3600, 'interval': 20}, 'concurrent_tasks': [{'type': 'search', 'weight': 1, 'params': {'nq': 1, 'top_k': 1, 'search_param': {'ef': 16}, 'random_data': True}}]} (params_check.py:31)

[yanliang567]

/assign @czs007 /unassign

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[xiaofan-luan]

is this still a issue?