search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.32k stars 2.91k forks source link

[Bug]: [benchmark][cluster] Occasional `DropCollection` reports an error `context deadline exceeded` #26930

Closed wangting0128 closed 1 year ago

wangting0128 commented 1 year ago

Is there an existing issue for this?

Environment

- Milvus version:2.3.0-20230907-264c542b
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.0.dev36
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-concurrent-h4hnc client pod: fouramf-concurrent-h4hnc-4015192404

server:

[2023-09-08 05:16:39,344 -  INFO - fouram]: [Base] Deploy initial state: 
I0907 13:42:37.816197     388 request.go:665] Waited for 1.156306592s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/apps/v1?timeout=32s
NAME                                                              READY   STATUS      RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouram-75-1999-etcd-0                                             1/1     Running     0               2m13s   10.104.12.231   4am-node17   <none>           <none>
fouram-75-1999-etcd-1                                             1/1     Running     0               2m13s   10.104.5.189    4am-node12   <none>           <none>
fouram-75-1999-etcd-2                                             1/1     Running     0               2m13s   10.104.18.55    4am-node25   <none>           <none>
fouram-75-1999-kafka-0                                            1/1     Running     2 (93s ago)     2m13s   10.104.12.233   4am-node17   <none>           <none>
fouram-75-1999-kafka-1                                            1/1     Running     2 (86s ago)     2m13s   10.104.5.190    4am-node12   <none>           <none>
fouram-75-1999-kafka-2                                            1/1     Running     2 (91s ago)     2m13s   10.104.18.54    4am-node25   <none>           <none>
fouram-75-1999-milvus-datacoord-699477966b-tjs4c                  1/1     Running     0               2m13s   10.104.4.151    4am-node11   <none>           <none>
fouram-75-1999-milvus-datanode-576bfb5748-ndkwl                   1/1     Running     0               2m13s   10.104.6.101    4am-node13   <none>           <none>
fouram-75-1999-milvus-indexcoord-d65b596cf-6kv95                  1/1     Running     0               2m13s   10.104.4.152    4am-node11   <none>           <none>
fouram-75-1999-milvus-indexnode-55c685fbdd-c2d2h                  1/1     Running     0               2m13s   10.104.4.153    4am-node11   <none>           <none>
fouram-75-1999-milvus-proxy-7f4b6d9bc7-twjnl                      1/1     Running     0               2m13s   10.104.12.225   4am-node17   <none>           <none>
fouram-75-1999-milvus-querycoord-7c87799468-wp8z9                 1/1     Running     0               2m13s   10.104.4.149    4am-node11   <none>           <none>
fouram-75-1999-milvus-querynode-658d7b49c6-h9tx7                  1/1     Running     0               2m13s   10.104.4.150    4am-node11   <none>           <none>
fouram-75-1999-milvus-querynode-658d7b49c6-pntqq                  1/1     Running     0               2m13s   10.104.6.102    4am-node13   <none>           <none>
fouram-75-1999-milvus-rootcoord-68d5559cc4-qhh2k                  1/1     Running     0               2m13s   10.104.12.226   4am-node17   <none>           <none>
fouram-75-1999-minio-0                                            1/1     Running     0               2m13s   10.104.5.185    4am-node12   <none>           <none>
fouram-75-1999-minio-1                                            1/1     Running     0               2m13s   10.104.4.157    4am-node11   <none>           <none>
fouram-75-1999-minio-2                                            1/1     Running     0               2m13s   10.104.12.236   4am-node17   <none>           <none>
fouram-75-1999-minio-3                                            1/1     Running     0               2m13s   10.104.18.59    4am-node25   <none>           <none> (base.py:221)
[2023-09-08 05:16:39,344 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'STATUS|fouram-75-1999-milvus|fouram-75-1999-minio|fouram-75-1999-etcd|fouram-75-1999-pulsar|fouram-75-1999-kafka'  (util_cmd.py:14)
[2023-09-08 05:16:48,763 -  INFO - fouram]: [CliClient] pod details of release(fouram-75-1999): 
 I0908 05:16:40.621173     528 request.go:665] Waited for 1.164153783s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/monitoring.coreos.com/v1?timeout=32s
NAME                                                              READY   STATUS        RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouram-75-1999-etcd-0                                             1/1     Running       1 (7h31m ago)   15h     10.104.12.231   4am-node17   <none>           <none>
fouram-75-1999-etcd-1                                             1/1     Running       0               15h     10.104.5.189    4am-node12   <none>           <none>
fouram-75-1999-etcd-2                                             1/1     Running       0               15h     10.104.18.55    4am-node25   <none>           <none>
fouram-75-1999-kafka-0                                            1/1     Running       2 (15h ago)     15h     10.104.12.233   4am-node17   <none>           <none>
fouram-75-1999-kafka-1                                            1/1     Running       2 (15h ago)     15h     10.104.5.190    4am-node12   <none>           <none>
fouram-75-1999-kafka-2                                            1/1     Running       2 (15h ago)     15h     10.104.18.54    4am-node25   <none>           <none>
fouram-75-1999-milvus-datacoord-699477966b-tjs4c                  1/1     Running       0               15h     10.104.4.151    4am-node11   <none>           <none>
fouram-75-1999-milvus-datanode-576bfb5748-ndkwl                   1/1     Running       0               15h     10.104.6.101    4am-node13   <none>           <none>
fouram-75-1999-milvus-indexcoord-d65b596cf-6kv95                  1/1     Running       0               15h     10.104.4.152    4am-node11   <none>           <none>
fouram-75-1999-milvus-indexnode-55c685fbdd-c2d2h                  1/1     Running       0               15h     10.104.4.153    4am-node11   <none>           <none>
fouram-75-1999-milvus-proxy-7f4b6d9bc7-twjnl                      1/1     Running       0               15h     10.104.12.225   4am-node17   <none>           <none>
fouram-75-1999-milvus-querycoord-7c87799468-wp8z9                 1/1     Running       0               15h     10.104.4.149    4am-node11   <none>           <none>
fouram-75-1999-milvus-querynode-658d7b49c6-h9tx7                  1/1     Running       0               15h     10.104.4.150    4am-node11   <none>           <none>
fouram-75-1999-milvus-querynode-658d7b49c6-pntqq                  1/1     Running       0               15h     10.104.6.102    4am-node13   <none>           <none>
fouram-75-1999-milvus-rootcoord-68d5559cc4-qhh2k                  1/1     Running       0               15h     10.104.12.226   4am-node17   <none>           <none>
fouram-75-1999-minio-0                                            1/1     Running       0               15h     10.104.5.185    4am-node12   <none>           <none>
fouram-75-1999-minio-1                                            1/1     Running       0               15h     10.104.4.157    4am-node11   <none>           <none>
fouram-75-1999-minio-2                                            1/1     Running       0               15h     10.104.12.236   4am-node17   <none>           <none>
fouram-75-1999-minio-3                                            1/1     Running       0               15h     10.104.18.59    4am-node25   <none>           <none>

client error:

[2023-09-07 20:03:59,403 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2023-09-07 20:03:59,403 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2023-09-07 20:03:59,403 -  INFO - fouram]: grpc     load                                                                             329     0(0.00%) |     33       3    2983      5 |    0.20        0.00 (stats.py:789)
[2023-09-07 20:03:59,403 -  INFO - fouram]: grpc     query                                                                           3412     0(0.00%) |     19       4    1486      6 |    1.00        0.00 (stats.py:789)
[2023-09-07 20:03:59,403 -  INFO - fouram]: grpc     scene_test                                                                       644     0(0.00%) | 311514   63853 8002345  65000 |    0.30        0.00 (stats.py:789)
[2023-09-07 20:03:59,403 -  INFO - fouram]: grpc     search                                                                          6724     0(0.00%) |     57      19    1268     38 |    2.30        0.00 (stats.py:789)
[2023-09-07 20:03:59,403 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2023-09-07 20:03:59,403 -  INFO - fouram]:          Aggregated                                                                     11109     0(0.00%) |  18100       3 8002345     26 |    3.80        0.00 (stats.py:789)
[2023-09-07 20:03:59,403 -  INFO - fouram]:  (stats.py:790)
[2023-09-07 20:03:59,404 -  INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2023-09-07 20:03:59,404 -  INFO - fouram]: Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs (stats.py:819)
[2023-09-07 20:03:59,404 -  INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2023-09-07 20:03:59,404 -  INFO - fouram]: grpc     load                                                                                    5      6      7      8     16     41    220    810   3000   3000   3000    329 (stats.py:819)
[2023-09-07 20:03:59,405 -  INFO - fouram]: grpc     query                                                                                   6      7      8      9     12     34    190    400   1200   1500   1500   3412 (stats.py:819)
[2023-09-07 20:03:59,405 -  INFO - fouram]: grpc     scene_test                                                                          65000  66000  66000  66000  67000  68000 7986000 7993000 8002000 8002000 8002000    644 (stats.py:819)
[2023-09-07 20:03:59,405 -  INFO - fouram]: grpc     search                                                                                 38     55     64     67     78    100    280    510   1100   1300   1300   6724 (stats.py:819)
[2023-09-07 20:03:59,405 -  INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2023-09-07 20:03:59,405 -  INFO - fouram]:          Aggregated                                                                             26     43     60     66     98  65000  66000  66000 7987000 8001000 8002000  11109 (stats.py:819)
[2023-09-07 20:03:59,405 -  INFO - fouram]:  (stats.py:820)
[2023-09-07 20:04:04,720 - ERROR - fouram]: RPC error: [drop_collection], <MilvusException: (code=1, message=context deadline exceeded)>, <Time:{'RPC start': '2023-09-07 20:03:54.717784', 'RPC error': '2023-09-07 20:04:04.720271'}> (decorators.py:108)
[2023-09-07 20:04:04,722 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=context deadline exceeded)> (api_request.py:53)
[2023-09-07 20:04:04,722 - ERROR - fouram]: [CheckFunc] drop_collection request check failed, response:<MilvusException: (code=1, message=context deadline exceeded)> (func_check.py:52)
[2023-09-07 20:04:04,724 - ERROR - fouram]: [func_time_catch] :  (api_request.py:120)
[2023-09-07 20:04:19,405 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2023-09-07 20:04:19,406 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2023-09-07 20:04:19,406 -  INFO - fouram]: grpc     load                                                                             333     0(0.00%) |     32       3    2983      5 |    0.20        0.00 (stats.py:789)
[2023-09-07 20:04:19,406 -  INFO - fouram]: grpc     query                                                                           3457     0(0.00%) |     19       4    1486      6 |    0.90        0.00 (stats.py:789)
[2023-09-07 20:04:19,406 -  INFO - fouram]: grpc     scene_test                                                                       655     1(0.15%) | 307430   63853 8002345  65000 |    0.30        0.00 (stats.py:789)
[2023-09-07 20:04:19,406 -  INFO - fouram]: grpc     search                                                                          6798     0(0.00%) |     57      19    1268     38 |    2.20        0.00 (stats.py:789)
[2023-09-07 20:04:19,406 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2023-09-07 20:04:19,406 -  INFO - fouram]:          Aggregated                                                                     11243     1(0.01%) |  17952       3 8002345     26 |    3.60        0.00 (stats.py:789)
[2023-09-07 20:04:19,406 -  INFO - fouram]:  (stats.py:790)
[2023-09-07 20:04:19,407 -  INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2023-09-07 20:04:19,407 -  INFO - fouram]: Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs (stats.py:819)
[2023-09-07 20:04:19,407 -  INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2023-09-07 20:04:19,407 -  INFO - fouram]: grpc     load                                                                                    5      6      7      8     16     41    220    810   3000   3000   3000    333 (stats.py:819)
[2023-09-07 20:04:19,407 -  INFO - fouram]: grpc     query                                                                                   6      7      8      9     12     41    190    400   1200   1500   1500   3457 (stats.py:819)
[2023-09-07 20:04:19,407 -  INFO - fouram]: grpc     scene_test                                                                          65000  66000  66000  66000  67000  68000 7984000 7993000 8002000 8002000 8002000    655 (stats.py:819)
[2023-09-07 20:04:19,408 -  INFO - fouram]: grpc     search                                                                                 38     56     64     68     80    110    280    510   1100   1300   1300   6798 (stats.py:819)
[2023-09-07 20:04:19,408 -  INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2023-09-07 20:04:19,408 -  INFO - fouram]:          Aggregated                                                                             26     44     60     67    100  65000  66000  66000 7987000 8001000 8002000  11243 (stats.py:819)
[2023-09-07 20:04:19,408 -  INFO - fouram]:  (stats.py:820)

test result:

{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c2m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '4.0',
                                                              'memory': '64Gi'},
                                                   'requests': {'cpu': '3.0',
                                                                'memory': '33Gi'}},
                                     'replicas': 2},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0',
                                                              'memory': '16Gi'},
                                                   'requests': {'cpu': '5.0',
                                                                'memory': '9Gi'}},
                                     'replicas': 1},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0',
                                                             'memory': '4Gi'},
                                                  'requests': {'cpu': '2.0',
                                                               'memory': '3Gi'}},
                                    'replicas': 1},
                       'cluster': {'enabled': True},
                       'pulsar': {'bookkeeper': {'volumes': {'journal': {'storageClassName': 'local-path'},
                                                             'ledgers': {'storageClassName': 'local-path'}}},
                                  'zookeeper': {'volumes': {'data': {'storageClassName': 'local-path'}}},
                                  'enabled': False},
                       'kafka': {'persistence': {'storageClass': 'local-path'},
                                 'enabled': True},
                       'minio': {'persistence': {'storageClass': 'local-path'},
                                 'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'global': {'storageClass': 'local-path'},
                                'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': '2.3.0-20230907-264c542b'}}},
            'host': 'fouram-75-1999-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_100m_hnsw_ddl_dql_filter_kafka_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 100000000,
                                                    'ni_per': 50000},
                                 'collection_params': {'other_fields': ['float_1'],
                                                       'shards_num': 2},
                                 'load_params': {},
                                 'query_params': {},
                                 'search_params': {},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False,
                                                          'reset_db': False},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 8,
                                                                  'efConstruction': 200}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '12h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 20,
                                                       'params': {'nq': 10,
                                                                  'top_k': 10,
                                                                  'search_param': {'ef': 16},
                                                                  'expr': {'float_1': {'GT': -1.0,
                                                                                       'LT': 50000000.0}},
                                                                  'guarantee_timestamp': None,
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'timeout': 60,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 10,
                                                       'params': {'ids': [0,
                                                                          1,
                                                                          2,
                                                                          3,
                                                                          4,
                                                                          5,
                                                                          6,
                                                                          7,
                                                                          8,
                                                                          9],
                                                                  'expr': None,
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'timeout': 60}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1,
                                                                  'timeout': 30}},
                                                      {'type': 'scene_test',
                                                       'weight': 2,
                                                       'params': {'dim': 128,
                                                                  'data_size': 3000,
                                                                  'nb': 3000,
                                                                  'index_type': 'IVF_SQ8',
                                                                  'index_param': {'nlist': 2048},
                                                                  'metric_type': 'L2'}}]},
            'run_id': 2023090740301991,
            'datetime': '2023-09-07 13:40:30.007783',
            'client_version': '2.3'},
 'result': {'test_result': {'index': {'RT': 7585.5023},
                            'insert': {'total_time': 3708.6757,
                                       'VPS': 26963.8027,
                                       'batch_time': 1.8543,
                                       'batch': 50000},
                            'flush': {'RT': 3.029},
                            'load': {'RT': 195.2074},
                            'Locust': {'Aggregated': {'Requests': 174618,
                                                      'Fails': 1,
                                                      'RPS': 4.04,
                                                      'fail_s': 0.0,
                                                      'RT_max': 8002345.08,
                                                      'RT_avg': 4942.82,
                                                      'TP50': 24,
                                                      'TP99': 66000.0},
                                       'load': {'Requests': 5328,
                                                'Fails': 0,
                                                'RPS': 0.12,
                                                'fail_s': 0.0,
                                                'RT_max': 2983.42,
                                                'RT_avg': 14.31,
                                                'TP50': 4,
                                                'TP99': 240.0},
                                       'query': {'Requests': 52892,
                                                 'Fails': 0,
                                                 'RPS': 1.22,
                                                 'fail_s': 0.0,
                                                 'RT_max': 1486.09,
                                                 'RT_avg': 8.08,
                                                 'TP50': 6,
                                                 'TP99': 54},
                                       'scene_test': {'Requests': 10625,
                                                      'Fails': 1,
                                                      'RPS': 0.25,
                                                      'fail_s': 0.0,
                                                      'RT_max': 8002345.08,
                                                      'RT_avg': 80753.02,
                                                      'TP50': 65000.0,
                                                      'TP99': 72000.0},
                                       'search': {'Requests': 105773,
                                                  'Fails': 0,
                                                  'RPS': 2.45,
                                                  'fail_s': 0.0,
                                                  'RT_max': 1268.94,
                                                  'RT_avg': 43.49,
                                                  'TP50': 34,
                                                  'TP99': 150.0}}}}}

Expected Behavior

No response

Steps To Reproduce

1. deploy a Cluster Milvus
2. prepare 100m data
3. concurrent request: load、query、search、scene_test 《- scene_test raise drop collection error

Milvus Log

Milvus log: {pod=~"fouram-75-1999-milvus-.*"} |~ "517c8011dfe8cdbc58cd4ce5b180a69f"

截屏2023-09-08 15 10 28 
  |   | proxyproxy4am-node17fouram-75-1999-milvus-proxy-7f4b6d9bc7-twjnl | [2023/09/07 20:04:04.720 +00:00] [DEBUG] [proxy/impl.go:406] ["DropCollection done"] [traceID=517c8011dfe8cdbc58cd4ce5b180a69f] [role=proxy] [db=default] [collection=fouram_VffHcdiq] [BeginTs=444102615902453764] [EndTs=444102615902453764]
-- | -- | -- | --
  |   | rootcoordrootcoord4am-node17fouram-75-1999-milvus-rootcoord-68d5559cc4-qhh2k | [2023/09/07 20:04:04.719 +00:00] [INFO] [rootcoord/root_coord.go:976] ["failed to drop collection"] [traceID=517c8011dfe8cdbc58cd4ce5b180a69f] [role=rootcoord] [error="context deadline exceeded"] [name=fouram_VffHcdiq] [ts=444102615902453765]
  |   | rootcoordrootcoord4am-node17fouram-75-1999-milvus-rootcoord-68d5559cc4-qhh2k | [2023/09/07 20:04:04.719 +00:00] [ERROR] [rootcoord/redo.go:63] ["failed to execute step"] [error="context deadline exceeded"] [desc="change collection state, collection: 444096610386780832, ts: 444102615902453765, state: CollectionDropping"] [stack="github.com/milvus-io/milvus/internal/rootcoord.(*baseRedoTask).Execute\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/redo.go:63\ngithub.com/milvus-io/milvus/internal/rootcoord.(*dropCollectionTask).Execute\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/drop_collection_task.go:118\ngithub.com/milvus-io/milvus/internal/rootcoord.(*scheduler).execute\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/scheduler.go:88\ngithub.com/milvus-io/milvus/internal/rootcoord.(*scheduler).taskLoop\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/scheduler.go:99"]
  |   | querycoordquerycoord4am-node11fouram-75-1999-milvus-querycoord-7c87799468-wp8z9 | [2023/09/07 20:04:00.987 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] ["retry time"=0] [error="stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:117 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:349 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetRecoveryInfoV2\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:151 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetRecoveryInfoV2\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:191 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).PullNextTargetV2.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:220 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).PullNextTargetV2\n/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:101 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).UpdateCollectionNextTarget: rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [errorVerbose="stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  \| github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  \| \t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408\n  \| github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  \| \t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422\n  \| github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  \| \t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:117\n  \| github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetRecoveryInfoV2\n  \| \t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:349\n  \| github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetRecoveryInfoV2\n  \| \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:151\n  \| github.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).PullNextTargetV2.func1\n  \| \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:191\n  \| github.com/milvus-io/milvus/pkg/util/retry.Do\n  \| \t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\n  \| github.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).PullNextTargetV2\n  \| \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:220\n  \| github.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).UpdateCollectionNextTarget\n  \| \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:101\n  \| github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).updateNextTarget\n  \| \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:228\n  \| github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).check\n  \| \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:157\n  \| github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).tryUpdateTarget\n  \| \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:189\n  \| github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).schedule\n  \| \t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:104\n  \| runtime.goexit\n  \| \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  \| /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  \| /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  \| /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:117 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n  \| /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:349 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetRecoveryInfoV2\n  \| /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:151 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetRecoveryInfoV2\n  \| /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:191 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).PullNextTargetV2.func1\n  \| /go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n  \| /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:220 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).PullNextTargetV2\n  \| /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:101 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).UpdateCollectionNextTarget\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).PullNextTargetV2\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:220\ngithub.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).UpdateCollectionNextTarget\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:101\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).updateNextTarget\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:228\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).check\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:157\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).tryUpdateTarget\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:189\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:104"]
  |   | datacoorddatacoord4am-node11fouram-75-1999-milvus-datacoord-699477966b-tjs4c | [2023/09/07 20:04:00.987 +00:00] [ERROR] [datacoord/services.go:773] ["get collection info from rootcoord failed"] [traceID=77b513531f27a8d7ab0a1e57b16b6969] [collectionID=444096610251047121] [partitionIDs="[]"] [error="stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:208 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).describeCollectionInternal\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:214 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollectionInternal\n/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:58 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).DescribeCollectionInternal\n/go/src/github.com/milvus-io/milvus/internal/datacoord/services.go:771 github.com/milvus-io/milvus/internal/datacoord.(*Server).GetRecoveryInfoV2\n/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:308 github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).GetRecoveryInfoV2\n/go/src/github.com/milvus-io/milvus/internal/proto/datapb/data_coord.pb.go:6284 github.com/milvus-io/milvus/internal/proto/datapb._DataCoord_GetRecoveryInfoV2_Handler.func1: rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [errorVerbose="stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n  -- stack trace:\n  \| github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  \| \t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408\n  \| github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  \| \t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422\n  \| github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  \| \t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\n  \| github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).describeCollectionInternal\n  \| \t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:208\n  \| github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollectionInternal\n  \| \t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:214\n  \| github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).DescribeCollectionInternal\n  \| \t/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:58\n  \| github.com/milvus-io/milvus/internal/datacoord.(*Server).GetRecoveryInfoV2\n  \| \t/go/src/github.com/milvus-io/milvus/internal/datacoord/services.go:771\n  \| github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).GetRecoveryInfoV2\n  \| \t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:308\n  \| github.com/milvus-io/milvus/internal/proto/datapb._DataCoord_GetRecoveryInfoV2_Handler.func1\n  \| \t/go/src/github.com/milvus-io/milvus/internal/proto/datapb/data_coord.pb.go:6284\n  \| github.com/milvus-io/milvus/pkg/util/interceptor.ServerIDValidationUnaryServerInterceptor.func1\n  \| \t/go/src/github.com/milvus-io/milvus/pkg/util/interceptor/server_id_interceptor.go:54\n  \| github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  \| \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n  \| github.com/milvus-io/milvus/pkg/util/interceptor.ClusterValidationUnaryServerInterceptor.func1\n  \| \t/go/src/github.com/milvus-io/milvus/pkg/util/interceptor/cluster_interceptor.go:48\n  \| github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  \| \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n  \| github.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n  \| \t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\n  \| github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  \| \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n  \| go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n  \| \t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.38.0/interceptor.go:342\n  \| github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n  \| \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\n  \| github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n  \| \t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\n  \| github.com/milvus-io/milvus/internal/proto/datapb._DataCoord_GetRecoveryInfoV2_Handler\n  \| \t/go/src/github.com/milvus-io/milvus/internal/proto/datapb/data_coord.pb.go:6286\n  \| google.golang.org/grpc.(*Server).processUnaryRPC\n  \| \t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\n  \| google.golang.org/grpc.(*Server).handleStream\n  \| \t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\n  \| google.golang.org/grpc.(*Server).serveStreams.func1.2\n  \| \t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966\n  \| runtime.goexit\n  \| \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n  \| /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  \| /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  \| /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  \| /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:208 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).describeCollectionInternal\n  \| /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:214 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollectionInternal\n  \| /go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:58 github.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).DescribeCollectionInternal\n  \| /go/src/github.com/milvus-io/milvus/internal/datacoord/services.go:771 github.com/milvus-io/milvus/internal/datacoord.(*Server).GetRecoveryInfoV2\n  \| /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:308 github.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).GetRecoveryInfoV2\n  \| /go/src/github.com/milvus-io/milvus/internal/proto/datapb/data_coord.pb.go:6284 github.com/milvus-io/milvus/internal/proto/datapb._DataCoord_GetRecoveryInfoV2_Handler.func1\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"] [stack="github.com/milvus-io/milvus/internal/datacoord.(*Server).GetRecoveryInfoV2\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/services.go:773\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).GetRecoveryInfoV2\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:308\ngithub.com/milvus-io/milvus/internal/proto/datapb._DataCoord_GetRecoveryInfoV2_Handler.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proto/datapb/data_coord.pb.go:6284\ngithub.com/milvus-io/milvus/pkg/util/interceptor.ServerIDValidationUnaryServerInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/interceptor/server_id_interceptor.go:54\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/interceptor.ClusterValidationUnaryServerInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/interceptor/cluster_interceptor.go:48\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.38.0/interceptor.go:342\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus/internal/proto/datapb._DataCoord_GetRecoveryInfoV2_Handler\n\t/go/src/github.com/milvus-io/milvus/internal/proto/datapb/data_coord.pb.go:6286\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"]
  |   | querycoordquerycoord4am-node11fouram-75-1999-milvus-querycoord-7c87799468-wp8z9 | [2023/09/07 20:04:00.987 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] ["retry time"=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:322\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:406\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:117\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetRecoveryInfoV2\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:349\ngithub.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetRecoveryInfoV2\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:151\ngithub.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).PullNextTargetV2.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:191\ngithub.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40\ngithub.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).PullNextTargetV2\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:220\ngithub.com/milvus-io/milvus/internal/querycoordv2/meta.(*TargetManager).UpdateCollectionNextTarget\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/target_manager.go:101\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).updateNextTarget\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:228\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).check\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:157\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).tryUpdateTarget\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:189\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).schedule\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/target_observer.go:104"]
  |   | querycoordquerycoord4am-node11fouram-75-1999-milvus-querycoord-7c87799468-wp8z9 | [2023/09/07 20:04:00.987 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] ["retry time"=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:322\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:406\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:196\ngithub.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetCollectionSchema\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:76\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*LeaderObserver).sync\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/leader_observer.go:254\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*LeaderObserver).observeCollection\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/leader_observer.go:127\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*LeaderObserver).observeSegmentsDist\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/leader_observer.go:104\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*LeaderObserver).observe\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/leader_observer.go:90\ngithub.com/milvus-io/milvus/internal/querycoordv2/observers.(*LeaderObserver).Start.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/observers/leader_observer.go:76"]
  |   | datacoorddatacoord4am-node11fouram-75-1999-milvus-datacoord-699477966b-tjs4c | [2023/09/07 20:04:00.986 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] ["retry time"=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:322\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:406\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).describeCollectionInternal\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:208\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollectionInternal\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:214\ngithub.com/milvus-io/milvus/internal/datacoord.(*CoordinatorBroker).DescribeCollectionInternal\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/coordinator_broker.go:58\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).GetRecoveryInfoV2\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/services.go:771\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).GetRecoveryInfoV2\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:308\ngithub.com/milvus-io/milvus/internal/proto/datapb._DataCoord_GetRecoveryInfoV2_Handler.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proto/datapb/data_coord.pb.go:6284\ngithub.com/milvus-io/milvus/pkg/util/interceptor.ServerIDValidationUnaryServerInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/interceptor/server_id_interceptor.go:54\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/interceptor.ClusterValidationUnaryServerInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/interceptor/cluster_interceptor.go:48\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.38.0/interceptor.go:342\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus/internal/proto/datapb._DataCoord_GetRecoveryInfoV2_Handler\n\t/go/src/github.com/milvus-io/milvus/internal/proto/datapb/data_coord.pb.go:6286\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"]
  |   | proxyproxy4am-node17fouram-75-1999-milvus-proxy-7f4b6d9bc7-twjnl | [2023/09/07 20:03:54.719 +00:00] [INFO] [proxy/impl.go:145] ["complete to invalidate collection meta cache"] [traceID=517c8011dfe8cdbc58cd4ce5b180a69f] [module=Proxy] [role=proxy] [db=default] [collectionName=] [collectionID=444096610386780832]
  |   | proxyproxy4am-node17fouram-75-1999-milvus-proxy-7f4b6d9bc7-twjnl | [2023/09/07 20:03:54.719 +00:00] [INFO] [proxy/impl.go:122] ["received request to invalidate collection meta cache"] [traceID=517c8011dfe8cdbc58cd4ce5b180a69f] [module=Proxy] [role=proxy] [db=default] [collectionName=] [collectionID=444096610386780832]
  |   | rootcoordrootcoord4am-node17fouram-75-1999-milvus-rootcoord-68d5559cc4-qhh2k | [2023/09/07 20:03:54.719 +00:00] [INFO] [rootcoord/root_coord.go:956] ["received request to drop collection"] [traceID=517c8011dfe8cdbc58cd4ce5b180a69f] [role=rootcoord] [dbName=default] [name=fouram_VffHcdiq]
  |   | proxyproxy4am-node17fouram-75-1999-milvus-proxy-7f4b6d9bc7-twjnl | [2023/09/07 20:03:54.718 +00:00] [DEBUG] [proxy/impl.go:392] ["DropCollection enqueued"] [traceID=517c8011dfe8cdbc58cd4ce5b180a69f] [role=proxy] [db=default] [collection=fouram_VffHcdiq] [BeginTs=444102615902453764] [EndTs=444102615902453764]
  |   | proxyproxy4am-node17fouram-75-1999-milvus-proxy-7f4b6d9bc7-twjnl | [2023/09/07 20:03:54.718 +00:00] [DEBUG] [proxy/impl.go:382] ["DropCollection received"] [traceID=517c8011dfe8cdbc58cd4ce5b180a69f] [role=proxy] [db=default] [collection=fouram_VffHcdiq]

Anything else?

No response

yanliang567 commented 1 year ago

/assign @elstic does this reproduce recently?

elstic commented 1 year ago

/assign @elstic does this reproduce recently?

this issue has not occurred recently