[Bug]: [benchmark][cluster] create partition failed `context deadline exceeded` in concurrent multi-partition scene

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version:2.4-20241025-d30e27e6-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.5rc7
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: multi-vector-corn-1-1729864800 test case name: test_hybrid_search_locust_dql_dml_partition_cluster

server:

NAME                                                              READY   STATUS      RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-corn-1-1729864800-1-etcd-0                           1/1     Running     0               3h20m   10.104.25.170   4am-node30   <none>           <none>
multi-vector-corn-1-1729864800-1-etcd-1                           1/1     Running     0               3h20m   10.104.30.138   4am-node38   <none>           <none>
multi-vector-corn-1-1729864800-1-etcd-2                           1/1     Running     0               3h20m   10.104.27.254   4am-node31   <none>           <none>
multi-vector-corn-1-1729864800-1-milvus-datanode-7d778d9f9xqhg9   1/1     Running     3 (3h15m ago)   3h20m   10.104.6.86     4am-node13   <none>           <none>
multi-vector-corn-1-1729864800-1-milvus-indexnode-65cf7fc9k98rl   1/1     Running     3 (3h19m ago)   3h20m   10.104.17.2     4am-node23   <none>           <none>
multi-vector-corn-1-1729864800-1-milvus-indexnode-65cf7fc9kgwcm   1/1     Running     3 (3h19m ago)   3h20m   10.104.32.14    4am-node39   <none>           <none>
multi-vector-corn-1-1729864800-1-milvus-indexnode-65cf7fc9lljgb   1/1     Running     3 (3h19m ago)   3h20m   10.104.13.102   4am-node16   <none>           <none>
multi-vector-corn-1-1729864800-1-milvus-indexnode-65cf7fc9p72xb   1/1     Running     3 (3h19m ago)   3h20m   10.104.9.182    4am-node14   <none>           <none>
multi-vector-corn-1-1729864800-1-milvus-mixcoord-84d49684bzz5kn   1/1     Running     3 (3h15m ago)   3h20m   10.104.6.85     4am-node13   <none>           <none>
multi-vector-corn-1-1729864800-1-milvus-proxy-7fb8489ddd-64qgf    1/1     Running     4 (3h15m ago)   3h20m   10.104.19.45    4am-node28   <none>           <none>
multi-vector-corn-1-1729864800-1-milvus-querynode-676fbfd7k9lfg   1/1     Running     0               3h20m   10.104.6.107    4am-node13   <none>           <none>
multi-vector-corn-1-1729864800-1-minio-0                          1/1     Running     0               3h20m   10.104.30.134   4am-node38   <none>           <none>
multi-vector-corn-1-1729864800-1-minio-1                          1/1     Running     0               3h20m   10.104.26.150   4am-node32   <none>           <none>
multi-vector-corn-1-1729864800-1-minio-2                          1/1     Running     0               3h20m   10.104.25.173   4am-node30   <none>           <none>
multi-vector-corn-1-1729864800-1-minio-3                          1/1     Running     0               3h20m   10.104.15.220   4am-node20   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-bookie-0                  1/1     Running     0               3h20m   10.104.21.193   4am-node24   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-bookie-1                  1/1     Running     0               3h20m   10.104.26.153   4am-node32   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-bookie-2                  1/1     Running     0               3h20m   10.104.25.174   4am-node30   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-bookie-init-zq94l         0/1     Completed   0               3h20m   10.104.1.245    4am-node10   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-broker-0                  1/1     Running     0               3h20m   10.104.23.203   4am-node27   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-proxy-0                   1/1     Running     0               3h20m   10.104.5.207    4am-node12   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-pulsar-init-np6gx         0/1     Completed   0               3h20m   10.104.19.46    4am-node28   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-recovery-0                1/1     Running     0               3h20m   10.104.19.47    4am-node28   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-zookeeper-0               1/1     Running     0               3h20m   10.104.30.135   4am-node38   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-zookeeper-1               1/1     Running     0               3h19m   10.104.15.232   4am-node20   <none>           <none>
multi-vector-corn-1-1729864800-1-pulsar-zookeeper-2               1/1     Running     0               3h18m   10.104.23.215   4am-node27   <none>           <none>

milvus log:

client log:

[2024-10-25 14:24:54,588 - DEBUG - fouram]: (api_request)  : [Partition] args: [<Collection>:
-------------
<name>: fouram_vZppYLpc
<description>: 
<schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}, {'name': 'float_vector_1', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}, {'name': 'float_vector_2', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}, {'name': 'float_vector_3', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}, {'name': 'int64_1', 'description': '', 'type': <DataType.INT64: 5>}, {'name': 'varchar_1', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 256}}], 'enable_dynamic_field': False}
, 'scene_test_partition_nyWaiOHG', ''], kwargs: {}, [requestId: e77bc5ee-92dc-11ef-b181-56c4858a6d9c] (api_request.py:77)
[2024-10-25 14:27:42,358 - ERROR - fouram]: (api_response) : [Partition] <MilvusException: (code=10001, message=context deadline exceeded)>, [requestId: e77bc5ee-92dc-11ef-b181-56c4858a6d9c] (api_request.py:57)

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `DQL & DML(partition)`
            verify concurrent DQL & DML(partition) scenario,
            which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 128dim,
                'float_vector_3': 128dim,
                scalar field: int64_1, varchar_1
            2. build indexes:
                IVF_FLAT: 'float_vector'
                HNSW: 'float_vector_1',
                DISKANN: 'float_vector_2'
                IVF_SQ8: 'float_vector_3'
                INVERTED: 'int64_1', 'varchar_1'
                default scalar index: 'id'
            3. insert 1 million data into 10 partitions
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 1
            7. concurrent request:
                - scene_test_partition
                    (partition: create->insert->flush->index again->load->search->release->search failed->drop)
                - search
                - hybrid_search
                - query

Milvus Log

No response

Anything else?

test result:

[2024-10-25 17:24:53,577 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-10-25 17:24:53,578 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-10-25 17:24:53,578 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-10-25 17:24:53,578 -  INFO - fouram]: grpc     hybrid_search                                                                   1347     0(0.00%) |  22584      22  123244  21000 |    0.12        0.00 (stats.py:789)
[2024-10-25 17:24:53,578 -  INFO - fouram]: grpc     query                                                                            152     0(0.00%) |  24145     164  178909  11000 |    0.01        0.00 (stats.py:789)
[2024-10-25 17:24:53,578 -  INFO - fouram]: grpc     scene_test_partition                                                             138     1(0.72%) | 840634  167769 1733266 791000 |    0.01        0.00 (stats.py:789)
[2024-10-25 17:24:53,578 -  INFO - fouram]: grpc     search                                                                          1261     0(0.00%) |  48550    7661  120731  49000 |    0.12        0.00 (stats.py:789)
[2024-10-25 17:24:53,578 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-10-25 17:24:53,578 -  INFO - fouram]:          Aggregated                                                                      2898     1(0.03%) |  72919      22 1733266  34000 |    0.27        0.00 (stats.py:789)
[2024-10-25 17:24:53,578 -  INFO - fouram]:  (stats.py:790)
[2024-10-25 17:24:53,581 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c8m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '32.0', 'memory': '64Gi'}, 'requests': {'cpu': '17.0', 'memory': '33Gi'}}, 'replicas': 1},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '8Gi'}, 'requests': {'cpu': '5.0', 'memory': '5Gi'}}, 'replicas': 4},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0', 'memory': '8Gi'}, 'requests': {'cpu': '2.0', 'memory': '5Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsar': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': '2.4-20241025-d30e27e6-amd64'}}},
            'host': 'multi-vector-corn-1-1729864800-1-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_dql_dml_partition_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {}, 'int64_1': {'index_type': 'INVERTED'}, 'varchar_1': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8, 'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'DISKANN', 'index_param': {}, 'metric_type': 'IP'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 2048},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_2': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_3': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}}},
                                                    'extra_partitions': {'partitions': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                        'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                        'partition_9'],
                                                                         'data_repeated': False},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 1000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1', 'float_vector_2', 'float_vector_3', 'int64_1', 'varchar_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20, 'during_time': '3h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'scene_test_partition',
                                                       'weight': 1,
                                                       'params': {'data_size': 3000,
                                                                  'ni': 3000,
                                                                  'nq': 1,
                                                                  'search_param': {'nprobe': 64},
                                                                  'limit': 1,
                                                                  'expr': None,
                                                                  'output_fields': ['*'],
                                                                  'guarantee_timestamp': None,
                                                                  'timeout': 600}},
                                                      {'type': 'search',
                                                       'weight': 8,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'nprobe': 1000},
                                                                  'expr': 'int64_1 >= 0',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                      'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                      'partition_9'],
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 8,
                                                       'params': {'nq': 1,
                                                                  'top_k': 100,
                                                                  'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
                                                                           {'search_param': {'ef': 64}, 'anns_field': 'float_vector_1', 'top_k': 10},
                                                                           {'search_param': {'search_list': 32}, 'anns_field': 'float_vector_2', 'top_k': 30},
                                                                           {'search_param': {'nprobe': 16}, 'anns_field': 'float_vector_3', 'top_k': 400}],
                                                                  'rerank': {'WeightedRanker': [0.85, 0.95, 0.51, 0.32]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                      'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                      'partition_9'],
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 > -1 && ',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                      'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                      'partition_9'],
                                                                  'timeout': 600,
                                                                  'consistency_level': None,
                                                                  'random_data': True,
                                                                  'random_count': 20,
                                                                  'random_range': [0, 100000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64',
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}}]},
            'run_id': 2024102550876129,
            'datetime': '2024-10-25 14:04:47.407653',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 94.8684,
                                      'float_vector_1': {'RT': 1.0175},
                                      'float_vector_2': {'RT': 7.0975},
                                      'float_vector_3': {'RT': 0.5917},
                                      'id': {'RT': 0.5146},
                                      'int64_1': {'RT': 0.5744},
                                      'varchar_1': {'RT': 0.515}},
                            'insert': {'total_time': 159.4096, 'VPS': 6300.7234, 'batch_time': 1.5941, 'batch': 10000.0},
                            'flush': {'RT': 2.6225},
                            'load': {'RT': 6.7008},
                            'Locust': {'Aggregated': {'Requests': 2898,
                                                      'Fails': 1,
                                                      'RPS': 0.27,
                                                      'fail_s': 0.0,
                                                      'RT_max': 1733266.44,
                                                      'RT_avg': 72919.64,
                                                      'TP50': 34000.0,
                                                      'TP99': 1050000.0},
                                       'hybrid_search': {'Requests': 1347,
                                                         'Fails': 0,
                                                         'RPS': 0.12,
                                                         'fail_s': 0.0,
                                                         'RT_max': 123244.24,
                                                         'RT_avg': 22584.26,
                                                         'TP50': 21000.0,
                                                         'TP99': 82000.0},
                                       'query': {'Requests': 152,
                                                 'Fails': 0,
                                                 'RPS': 0.01,
                                                 'fail_s': 0.0,
                                                 'RT_max': 178909.62,
                                                 'RT_avg': 24145.16,
                                                 'TP50': 12000.0,
                                                 'TP99': 133000.0},
                                       'scene_test_partition': {'Requests': 138,
                                                                'Fails': 1,
                                                                'RPS': 0.01,
                                                                'fail_s': 0.01,
                                                                'RT_max': 1733266.44,
                                                                'RT_avg': 840634.84,
                                                                'TP50': 793000.0,
                                                                'TP99': 1589000.0},
                                       'search': {'Requests': 1261,
                                                  'Fails': 0,
                                                  'RPS': 0.12,
                                                  'fail_s': 0.0,
                                                  'RT_max': 120731.16,
                                                  'RT_avg': 48550.71,
                                                  'TP50': 49000.0,
                                                  'TP99': 91000.0}}}}}

Look at the code, I found that for now even if just do create collection add partiton search on the partition on a multi node deployment you could see a bug here since meta cache is never update on adding partition case

@wangting0128 can you try to reproduce on a simpler case with 2 proxies?

Deploy milvus with two proxy and concurrent multi-partition (partition: create->insert->flush->index again->load->search->release->search failed->drop). This problem did not reoccur.