[Bug]: [benchmark][multi-replicas-loadbalance] queryNode panic and restart, client raise error when queryNode restarts and 10 minutes after restart

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version:master-20230725-a6808e64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.0.dev73
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server argo task: fouramf-dnkv9 clients argo task: fouramf-concurrent-xfr27

server:

NAME                                                              READY   STATUS             RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-ivfsq8-scene-low-etcd-0                                   1/1     Running            0               13m     10.104.16.151   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-etcd-1                                   1/1     Running            0               13m     10.104.12.78    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-low-etcd-2                                   1/1     Running            0               13m     10.104.4.181    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-datacoord-57b456f5bb-fn76f        1/1     Running            0               13m     10.104.16.143   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-datanode-54d4c6b9b8-gkp7f         1/1     Running            1 (9m41s ago)   13m     10.104.23.174   4am-node27   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-datanode-54d4c6b9b8-skpnr         1/1     Running            1 (9m41s ago)   13m     10.104.1.206    4am-node10   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-indexcoord-6569568ff4-knhh2       1/1     Running            0               13m     10.104.20.133   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-indexnode-5c944687b8-hjjxk        1/1     Running            0               13m     10.104.13.128   4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-indexnode-5c944687b8-ncsmp        1/1     Running            0               13m     10.104.19.171   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-indexnode-5c944687b8-r9wj9        1/1     Running            0               13m     10.104.5.192    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-proxy-6997c8857b-c66kz            1/1     Running            1 (9m41s ago)   13m     10.104.16.141   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-querycoord-77656688d5-fhxh2       1/1     Running            1 (9m41s ago)   13m     10.104.16.144   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-querynode-84b4978657-hdp66        1/1     Running            0               13m     10.104.18.235   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-querynode-84b4978657-sqb4f        1/1     Running            0               13m     10.104.4.187    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-querynode-84b4978657-tdkxr        1/1     Running            0               13m     10.104.5.201    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-rootcoord-768564f4f7-xkcdz        1/1     Running            1 (9m40s ago)   13m     10.104.5.193    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-low-minio-0                                  1/1     Running            0               13m     10.104.16.147   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-minio-1                                  1/1     Running            0               13m     10.104.4.179    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-low-minio-2                                  1/1     Running            0               13m     10.104.12.77    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-low-minio-3                                  1/1     Running            0               13m     10.104.6.50     4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-bookie-0                          1/1     Running            0               13m     10.104.24.148   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-bookie-1                          1/1     Running            0               13m     10.104.16.152   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-bookie-2                          1/1     Running            0               13m     10.104.20.139   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-bookie-init-gsc24                 0/1     Completed          0               13m     10.104.16.142   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-broker-0                          1/1     Running            0               13m     10.104.13.125   4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-proxy-0                           1/1     Running            0               13m     10.104.20.134   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-pulsar-init-55nsh                 0/1     Completed          0               13m     10.104.23.173   4am-node27   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-recovery-0                        1/1     Running            0               13m     10.104.5.191    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-zookeeper-0                       1/1     Running            0               13m     10.104.5.198    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-zookeeper-1                       1/1     Running            0               12m     10.104.1.212    4am-node10   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-zookeeper-2                       1/1     Running            0               12m     10.104.16.154   4am-node21   <none>           <none>

NAME                                                              READY   STATUS                   RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-ivfsq8-scene-low-etcd-0                                   1/1     Running                  0               3h52m   10.104.16.151   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-etcd-1                                   1/1     Running                  0               3h52m   10.104.12.78    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-low-etcd-2                                   1/1     Running                  0               3h52m   10.104.4.181    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-datacoord-57b456f5bb-fn76f        1/1     Running                  0               3h52m   10.104.16.143   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-datanode-54d4c6b9b8-gkp7f         1/1     Running                  1 (3h48m ago)   3h52m   10.104.23.174   4am-node27   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-datanode-54d4c6b9b8-skpnr         1/1     Running                  1 (3h48m ago)   3h52m   10.104.1.206    4am-node10   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-indexcoord-6569568ff4-knhh2       1/1     Running                  0               3h52m   10.104.20.133   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-indexnode-5c944687b8-hjjxk        1/1     Running                  0               3h52m   10.104.13.128   4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-indexnode-5c944687b8-ncsmp        1/1     Running                  0               3h52m   10.104.19.171   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-indexnode-5c944687b8-r9wj9        1/1     Running                  0               3h52m   10.104.5.192    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-proxy-6997c8857b-c66kz            1/1     Running                  1 (3h48m ago)   3h52m   10.104.16.141   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-querycoord-77656688d5-fhxh2       1/1     Running                  1 (3h48m ago)   3h52m   10.104.16.144   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-querynode-84b4978657-hdp66        1/1     Running                  0               3h52m   10.104.18.235   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-querynode-84b4978657-sqb4f        1/1     Running                  1 (163m ago)    3h52m   10.104.4.187    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-querynode-84b4978657-tdkxr        1/1     Running                  0               3h52m   10.104.5.201    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-low-milvus-rootcoord-768564f4f7-xkcdz        1/1     Running                  1 (3h48m ago)   3h52m   10.104.5.193    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-low-minio-0                                  1/1     Running                  0               3h52m   10.104.16.147   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-minio-1                                  1/1     Running                  0               3h52m   10.104.4.179    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-low-minio-2                                  1/1     Running                  0               3h52m   10.104.12.77    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-low-minio-3                                  1/1     Running                  0               3h52m   10.104.6.50     4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-bookie-0                          1/1     Running                  0               3h52m   10.104.24.148   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-bookie-1                          1/1     Running                  0               3h52m   10.104.16.152   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-bookie-2                          1/1     Running                  0               3h52m   10.104.20.139   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-bookie-init-gsc24                 0/1     Completed                0               3h52m   10.104.16.142   4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-broker-0                          1/1     Running                  0               3h52m   10.104.13.125   4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-proxy-0                           1/1     Running                  0               3h52m   10.104.20.134   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-pulsar-init-55nsh                 0/1     Completed                0               3h52m   10.104.23.173   4am-node27   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-recovery-0                        1/1     Running                  0               3h52m   10.104.5.191    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-zookeeper-0                       1/1     Running                  0               3h52m   10.104.5.198    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-zookeeper-1                       1/1     Running                  0               3h51m   10.104.1.212    4am-node10   <none>           <none>
lb-helm-ivfsq8-scene-low-pulsar-zookeeper-2                       1/1     Running                  0               3h50m   10.104.16.154   4am-node21   <none>           <none>

lb-helm-ivfsq8-scene-low-milvus-querynode-84b4978657-sqb4f log:

2023-07-25 07:25:45 | Version:   a6808e64
-- | --
  |   | 2023-07-25 07:25:45 | Welcome to use Milvus!

clients log:

[2023-07-25 07:35:58,587 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443093106640975157 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:55.450050', 'RPC error': '2023-07-25 07:35:58.587036'}> (decorators.py:108)
--
  |   | [2023-07-25 07:35:58,586 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443093106640975157 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:55.481546', 'RPC error': '2023-07-25 07:35:58.586905'}> (decorators.py:108)
  |   | [2023-07-25 07:35:58,586 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443093106640975157 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:35:58.583847', 'RPC error': '2023-07-25 07:35:58.586689'}> (decorators.py:108)
  |   | [2023-07-25 07:35:58,585 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443093106640975130 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:54.429852', 'RPC error': '2023-07-25 07:35:58.585267'}> (decorators.py:108)
  |   | [2023-07-25 07:35:58,585 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443093106640975130 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:54.475117', 'RPC error': '2023-07-25 07:35:58.585040'}> (decorators.py:108)
  |   | [2023-07-25 07:35:58,585 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443093106640975162 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:54.946488', 'RPC error': '2023-07-25 07:35:58.585084'}> (decorators.py:108)
  |   | [2023-07-25 07:35:58,584 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443093106640975162 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:54.983679', 'RPC error': '2023-07-25 07:35:58.584945'}> (decorators.py:108)
  |   | [2023-07-25 07:35:58,584 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443093106640975162 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:35:58.583303', 'RPC error': '2023-07-25 07:35:58.584601'}> (decorators.py:108)
  |   | [2023-07-25 07:35:58,584 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443093106640975130 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:35:58.582893', 'RPC error': '2023-07-25 07:35:58.584331'}> (decorators.py:108)
  |   | [2023-07-25 07:35:58,079 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443093106640975135 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:54.631336', 'RPC error': '2023-07-25 07:35:58.079566'}> (decorators.py:108)
  |   | [2023-07-25 07:35:58,079 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443093106640975135 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:54.670975', 'RPC error': '2023-07-25 07:35:58.079363'}> (decorators.py:108)
  |   | [2023-07-25 07:35:58,078 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443093106640975135 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:35:58.077168', 'RPC error': '2023-07-25 07:35:58.078728'}> (decorators.py:108)
  |   | [2023-07-25 07:35:56,076 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443093106640974758 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:52.579619', 'RPC error': '2023-07-25 07:35:56.076021'}> (decorators.py:108)
  |   | [2023-07-25 07:35:56,075 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443093106640974758 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:52.634702', 'RPC error': '2023-07-25 07:35:56.075911'}> (decorators.py:108)
  |   | [2023-07-25 07:35:56,075 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443093106640974758 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:35:56.068826', 'RPC error': '2023-07-25 07:35:56.075752'}> (decorators.py:108)
  |   | [2023-07-25 07:35:56,073 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443093106640974652 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:52.067743', 'RPC error': '2023-07-25 07:35:56.073554'}> (decorators.py:108)
  |   | [2023-07-25 07:35:56,073 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443093106640974652 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:52.163432', 'RPC error': '2023-07-25 07:35:56.073433'}> (decorators.py:108)
  |   | [2023-07-25 07:35:56,073 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443093106640974652 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:35:56.068685', 'RPC error': '2023-07-25 07:35:56.073243'}> (decorators.py:108)
  |   | [2023-07-25 07:35:56,070 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443093106640974525 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:52.067167', 'RPC error': '2023-07-25 07:35:56.070355'}> (decorators.py:108)
  |   | [2023-07-25 07:35:56,070 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443093106640974525 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:25:52.103568', 'RPC error': '2023-07-25 07:35:56.070175'}> (decorators.py:108)
  |   | [2023-07-25 07:35:56,069 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443093106640974525 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-07-25 07:35:56.068176', 'RPC error': '2023-07-25 07:35:56.069536'}> (decorators.py:108)
  |   | [2023-07-25 07:26:03,355 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: fail to Query, QueryNode ID = 13, reason=failed to query channel, err=worker(7) query failed: collection=443093106520755610: collection not found: channel=by-dev-rootcoord-dml_6_443093106520755610v0: fail to access shard delegator: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:26:03.148865', 'RPC error': '2023-07-25 07:26:03.354990'}> (decorators.py:108)
  |   | [2023-07-25 07:26:03,148 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:03,148 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit: channel=by-dev-rootcoord-dml_6_443093106520755610v0: fail to access shard delegator: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:26:02.939541', 'RPC error': '2023-07-25 07:26:03.147580'}> (decorators.py:108)
  |   | [2023-07-25 07:26:03,147 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:02,938 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: fail to Query, QueryNode ID = 13, reason=failed to query channel, err=worker(7) query failed: collection=443093106520755610: collection not found: channel=by-dev-rootcoord-dml_6_443093106520755610v0: fail to access shard delegator: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:26:02.732861', 'RPC error': '2023-07-25 07:26:02.938497'}> (decorators.py:108)
  |   | [2023-07-25 07:26:02,732 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:02,732 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit: channel=by-dev-rootcoord-dml_6_443093106520755610v0: fail to access shard delegator: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:26:02.525236', 'RPC error': '2023-07-25 07:26:02.731790'}> (decorators.py:108)
  |   | [2023-07-25 07:26:02,731 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:02,524 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:02,524 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit: channel=by-dev-rootcoord-dml_6_443093106520755610v0: fail to access shard delegator: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:26:02.315541', 'RPC error': '2023-07-25 07:26:02.524018'}> (decorators.py:108)
  |   | [2023-07-25 07:26:02,524 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:02,314 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:02,314 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit: channel=by-dev-rootcoord-dml_6_443093106520755610v0: fail to access shard delegator: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:26:02.106840', 'RPC error': '2023-07-25 07:26:02.314266'}> (decorators.py:108)
  |   | [2023-07-25 07:26:02,314 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=13, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:02,106 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:02,106 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit: channel=by-dev-rootcoord-dml_7_443093106520755610v1: fail to access shard delegator: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:26:01.899739', 'RPC error': '2023-07-25 07:26:02.105749'}> (decorators.py:108)
  |   | [2023-07-25 07:26:02,105 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:01,897 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: fail to Query, QueryNode ID = 13, reason=failed to query channel, err=worker(7) query failed: collection=443093106520755610: collection not found: channel=by-dev-rootcoord-dml_6_443093106520755610v0: fail to access shard delegator: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:26:01.691577', 'RPC error': '2023-07-25 07:26:01.897931'}> (decorators.py:108)
  |   | [2023-07-25 07:26:01,691 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:01,690 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit: channel=by-dev-rootcoord-dml_7_443093106520755610v1: fail to access shard delegator: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:26:01.482573', 'RPC error': '2023-07-25 07:26:01.690175'}> (decorators.py:108)
  |   | [2023-07-25 07:26:01,690 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:01,482 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:26:01,481 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit: channel=by-dev-rootcoord-dml_7_443093106520755610v1: fail to access shard delegator: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:26:01.275309', 'RPC error': '2023-07-25 07:26:01.481491'}> (decorators.py:108)
  |   | [2023-07-25 07:26:01,481 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = collection=443093106520755610: collection not loaded
  |   | [2023-07-25 07:25:57,190 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: fail to Query, QueryNode ID = 7, reason=target node id not match target id = 7, node id = 14: channel=by-dev-rootcoord-dml_1_443093106520955683v1: fail to access shard delegator: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:56.986475', 'RPC error': '2023-07-25 07:25:57.190919'}> (decorators.py:108)
  |   | [2023-07-25 07:25:50,910 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:50.700923', 'RPC error': '2023-07-25 07:25:50.910687'}> (decorators.py:108)
  |   | [2023-07-25 07:25:50,699 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:50.495675', 'RPC error': '2023-07-25 07:25:50.699812'}> (decorators.py:108)
  |   | [2023-07-25 07:25:50,494 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:50.168731', 'RPC error': '2023-07-25 07:25:50.494502'}> (decorators.py:108)
  |   | [2023-07-25 07:25:50,167 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:49.963737', 'RPC error': '2023-07-25 07:25:50.167716'}> (decorators.py:108)
  |   | [2023-07-25 07:25:50,073 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = stage=node id: 14 is unhealthy: service not ready
  |   | [2023-07-25 07:25:50,073 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = stage=node id: 14 is unhealthy: service not ready
  |   | /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit: channel=by-dev-rootcoord-dml_7_443093106520755610v1: fail to access shard delegator: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:49.866170', 'RPC error': '2023-07-25 07:25:50.072773'}> (decorators.py:108)
  |   | [2023-07-25 07:25:50,072 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = stage=node id: 14 is unhealthy: service not ready
  |   | [2023-07-25 07:25:49,962 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:49.758704', 'RPC error': '2023-07-25 07:25:49.962698'}> (decorators.py:108)
  |   | [2023-07-25 07:25:49,865 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = stage=node id: 14 is unhealthy: service not ready
  |   | [2023-07-25 07:25:49,865 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = stage=node id: 14 is unhealthy: service not ready
  |   | /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit: channel=by-dev-rootcoord-dml_7_443093106520755610v1: fail to access shard delegator: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:49.656866', 'RPC error': '2023-07-25 07:25:49.864911'}> (decorators.py:108)
  |   | [2023-07-25 07:25:49,864 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: rpc error: code = Unknown desc = stage=node id: 14 is unhealthy: service not ready
  |   | [2023-07-25 07:25:49,757 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:49.553859', 'RPC error': '2023-07-25 07:25:49.757429'}> (decorators.py:108)
  |   | [2023-07-25 07:25:49,692 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:49.489367', 'RPC error': '2023-07-25 07:25:49.692318'}> (decorators.py:108)
  |   | [2023-07-25 07:25:49,655 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: fail to Query, QueryNode ID = 5, reason=failed to query channel, err=worker(7) query failed: node id: 14 is unhealthy: service unavailable: channel=by-dev-rootcoord-dml_7_443093106520755610v1: fail to access shard delegator: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:49.450339', 'RPC error': '2023-07-25 07:25:49.655623'}> (decorators.py:108)
  |   | [2023-07-25 07:25:49,552 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:49.348649', 'RPC error': '2023-07-25 07:25:49.552787'}> (decorators.py:108)
  |   | [2023-07-25 07:25:49,488 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 14 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:49.349920', 'RPC error': '2023-07-25 07:25:49.488063'}> (decorators.py:108)
  |   | [2023-07-25 07:25:49,448 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 14 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:49.344260', 'RPC error': '2023-07-25 07:25:49.448178'}> (decorators.py:108)
  |   | [2023-07-25 07:25:49,347 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:49.144145', 'RPC error': '2023-07-25 07:25:49.347862'}> (decorators.py:108)
  |   | [2023-07-25 07:25:49,143 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:48.940173', 'RPC error': '2023-07-25 07:25:49.143272'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,939 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:48.735716', 'RPC error': '2023-07-25 07:25:48.939455'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,934 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:48.731028', 'RPC error': '2023-07-25 07:25:48.934411'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,733 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 14 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:48.680819', 'RPC error': '2023-07-25 07:25:48.733639'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,728 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:48.523471', 'RPC error': '2023-07-25 07:25:48.728908'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,720 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 14 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:48.679999', 'RPC error': '2023-07-25 07:25:48.719982'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,695 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 14 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:48.679335', 'RPC error': '2023-07-25 07:25:48.695632'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,522 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:48.299256', 'RPC error': '2023-07-25 07:25:48.522228'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,359 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 14 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:48.085941', 'RPC error': '2023-07-25 07:25:48.359338'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,325 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 14 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:48.085306', 'RPC error': '2023-07-25 07:25:48.325405'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,296 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 14 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:48.084959', 'RPC error': '2023-07-25 07:25:48.296702'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,241 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 14 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:48.084432', 'RPC error': '2023-07-25 07:25:48.241462'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,203 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 14 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:48.084092', 'RPC error': '2023-07-25 07:25:48.203339'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,108 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 0 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:48.083767', 'RPC error': '2023-07-25 07:25:48.108046'}> (decorators.py:108)
  |   | [2023-07-25 07:25:48,074 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=failed to load collection: stage=query node 0 is not ready: service not ready)>, <Time:{'RPC start': '2023-07-25 07:25:43.523888', 'RPC error': '2023-07-25 07:25:48.074175'}> (decorators.py:108)
  |   | /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit: channel=by-dev-rootcoord-dml_0_443093106520955683v0: fail to access shard delegator: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-07-25 07:25:43.281868', 'RPC error': '2023-07-25 07:25:47.848964'}> (decorators.py:108)
  |   | [2023-07-25 07:25:47,849 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to Search, QueryNode ID=5, reason=err: failed to connect 10.104.4.187:21123, reason: context deadline exceeded

Expected Behavior

No response

Steps To Reproduce

1、deploy cluster Milvus with 3 queryNodes
2、concurrent 9 clients which have 3 types: replica=1, replica=2 and replica=3; each type has 3 clients
   a. create a collection with shard_num=2
   b. insert 10m data, build IVF_SQ8 index
   c. load with replica=1 or 2 or 3
   d. concurrent query, search, scene_search_test by locust <- raise error

scene_search_test:
1. create a collection with dim=128
2. insert 3000 data
3. flush collection
4. count entities of collection
5. build IVF_SQ8 index with nlist=2048
6. load collection with replica=1 or 2 or 3
7. search collection once with nprobe=1028, nq=1, topk=10
8. drop collection

Milvus Log

No response

Anything else?

fouramf-server-lb-3qn-2dn-large:

    queryNode:
      resources:
        limits:
          cpu: '50.0'
          memory: 100Gi
        requests:
          cpu: '25.0'
          memory: 50Gi
      replicas: 3
    indexNode:
      resources:
        limits:
          cpu: '16.0'
          memory: 16Gi
        requests:
          cpu: '5.0'
          memory: 5Gi
      replicas: 3
    dataNode:
      replicas: 2
      resources:
        limits:
          cpu: '2.0'
          memory: 16Gi
        requests:
          cpu: '2.0'
          memory: 2Gi

fouramf-client-sift-ivfsq8-replica3-shard2-search-query-scene:

    load_params:
      replica_number: 3
    collection_params:
      shards_num: 2
    dataset_params:
      dim: 128
      dataset_name: sift
      dataset_size: 10m
      ni_per: 50000
      metric_type: L2
    index_params:
      index_type: IVF_SQ8
      index_param:
        nlist: 2048
    concurrent_params:
      concurrent_number: 3
      during_time: 2h
      interval: 20
    concurrent_tasks:
      - type: query
        weight: 1
        params:
          ids: [1, 100, 1000]
      - type: search
        weight: 1
        params:
          nq: 1
          top_k: 1
          search_param:
            nprobe: 64
          timeout: 60
          random_data: true
      - type: scene_search_test
        weight: 1
        params:
          replica_number: 3

fouramf-client-sift-ivfsq8-replica2-shard2-search-query-scene:

    load_params:
      replica_number: 2
    collection_params:
      shards_num: 2
    dataset_params:
      dim: 128
      dataset_name: sift
      dataset_size: 10m
      ni_per: 50000
      metric_type: L2
    index_params:
      index_type: IVF_SQ8
      index_param:
        nlist: 2048
    concurrent_params:
      concurrent_number: 3
      during_time: 2h
      interval: 20
    concurrent_tasks:
      - type: query
        weight: 1
        params:
          ids: [1, 100, 1000]
      - type: search
        weight: 1
        params:
          nq: 1
          top_k: 1
          search_param:
            nprobe: 64
          timeout: 60
          random_data: true
      - type: scene_search_test
        weight: 1
        params:
          replica_number: 2

fouramf-client-sift-ivfsq8-replica1-shard2-search-query-scene:

    load_params:
      replica_number: 1
    collection_params:
      shards_num: 2
    dataset_params:
      dim: 128
      dataset_name: sift
      dataset_size: 10m
      ni_per: 50000
      metric_type: L2
    index_params:
      index_type: IVF_SQ8
      index_param:
        nlist: 2048
    concurrent_params:
      concurrent_number: 3
      during_time: 2h
      interval: 20
    concurrent_tasks:
      - type: query
        weight: 1
        params:
          ids: [1, 100, 1000]
      - type: search
        weight: 1
        params:
          nq: 1
          top_k: 1
          search_param:
            nprobe: 64
          timeout: 60
          random_data: true
      - type: scene_search_test
        weight: 1
        params:
          replica_number: 1

@wangting0128 could you retry with #25895? The logs don't contain any information about this restart, the only thing we could see is the panic happened while loading, and this PR is a fix for loading

New recurring scene

server argo task: fouramf-6b6s8

deploy_config: fouramf-server-lb-3qn-2dn-large 
other_params: --milvus_tag=master-20230803-68ecf49e -k test_server_custom_parameters -s --deploy_mode=cluster --deploy_retain --release_name=lb-helm-ivfsq8-scene-high --update_helm_file

client argo task: fouramf-concurrent-vcg2d

other_params: --deploy_skip -s --release_name=lb-helm-ivfsq8-scene-high --case_skip_prepare_clean --case_skip_clean_collection
case_names: [ {"id": 1, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-ivfsq8-replica3-shard2-search-query-scene-high"}, {"id": 2, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-ivfsq8-replica2-shard2-search-query-scene-high"}, {"id": 3, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-ivfsq8-replica1-shard2-search-query-scene-high"}, {"id": 4, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-ivfsq8-replica3-shard2-search-query-scene-high"}, {"id": 5, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-ivfsq8-replica2-shard2-search-query-scene-high"}, {"id": 6, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-ivfsq8-replica1-shard2-search-query-scene-high"}, {"id": 7, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-ivfsq8-replica3-shard2-search-query-scene-high"}, {"id": 8, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-ivfsq8-replica2-shard2-search-query-scene-high"}, {"id": 9, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-ivfsq8-replica1-shard2-search-query-scene-high"} ]
test-sdk-version: 2.4.0.dev73

VF_SQ8 index，high concurrent request；
During the request process, the server component restarted multiple times due to undetermined reasons, resulting in request failure. The specific client-side error is as follows clients.log.zip

[2023-08-03 09:45:12,316 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443297752819115280 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 09:45:12.311972', 'RPC error': '2023-08-03 09:45:12.316325'}> (decorators.py:108)
[2023-08-03 09:45:12,316 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443297752819115280 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 09:23:47.435102', 'RPC error': '2023-08-03 09:45:12.316508'}> (decorators.py:108)
[2023-08-03 09:45:12,316 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443297752819115280 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 09:22:14.540197', 'RPC error': '2023-08-03 09:45:12.316614'}> (decorators.py:108)
[2023-08-03 09:45:12,316 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=collection 443297752819115280 has not been loaded to memory or load failed)> (api_request.py:53)
[2023-08-03 09:45:12,316 - ERROR - fouram]: [CheckFunc] load request check failed, response:<MilvusException: (code=1, message=collection 443297752819115280 has not been loaded to memory or load failed)> (func_check.py:52)
[2023-08-03 09:45:12,317 - ERROR - fouram]: [func_time_catch] :  (api_request.py:120)
[2023-08-03 09:45:16,389 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: fail to get shard leaders from QueryCoord: leader not available: node=23: node offline: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #1: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #2: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #3: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #4: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #5: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_7_443297752708831153v1: channel not available: context done during sleep after run#5: context deadline exceeded: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-08-03 09:45:06.384672', 'RPC error': '2023-08-03 09:45:16.389336'}> (decorators.py:108)
[2023-08-03 09:45:16,389 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: fail to get shard leaders from QueryCoord: leader not available: node=23: node offline: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #1: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #2: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #3: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #4: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #5: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_7_443297752708831153v1: channel not available: context done during sleep after run#5: context deadline exceeded: fail to search on all shard leaders)> (api_request.py:53)
[2023-08-03 09:45:16,389 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: fail to get shard leaders from QueryCoord: leader not available: node=23: node offline: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #1: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #2: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #3: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #4: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_6_443297752708831153v0: channel not available: attempt #5: fail to get shard leaders from QueryCoord: channel=channel not subscribed: channel lacks: channel=by-dev-rootcoord-dml_7_443297752708831153v1: channel not available: context done during sleep after run#5: context deadline exceeded: fail to search on all shard leaders)> (func_check.py:46)
[2023-08-03 09:45:16,396 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443297752819121523 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 09:45:12.346088', 'RPC error': '2023-08-03 09:45:16.396224'}> (decorators.py:108)
[2023-08-03 09:45:16,396 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443297752819121523 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 09:24:53.290265', 'RPC error': '2023-08-03 09:45:16.396463'}> (decorators.py:108)
[2023-08-03 09:45:16,396 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443297752819121523 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 09:23:07.353369', 'RPC error': '2023-08-03 09:45:16.396581'}> (decorators.py:108)
[2023-08-03 09:45:16,396 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=collection 443297752819121523 has not been loaded to memory or load failed)> (api_request.py:53)

server:

NAME                                                              READY   STATUS      RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-ivfsq8-scene-high-etcd-0                                  1/1     Running     0               4m2s    10.104.19.117   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-1                                  1/1     Running     0               4m2s    10.104.14.146   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-2                                  1/1     Running     0               4m1s    10.104.5.113    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datacoord-649884d8b9-bpqlg       1/1     Running     0               4m2s    10.104.18.131   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-59fb787b47-9vdlh        1/1     Running     0               4m2s    10.104.12.36    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-59fb787b47-l2kzp        1/1     Running     0               4m2s    10.104.1.82     4am-node10   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexcoord-7c79599795-nsghm      1/1     Running     0               4m2s    10.104.18.132   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-58b8888c45-dwr44       1/1     Running     0               4m2s    10.104.5.109    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-58b8888c45-fngbs       1/1     Running     0               4m2s    10.104.18.133   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-58b8888c45-n57rr       1/1     Running     0               4m2s    10.104.12.39    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-proxy-ff4bcc4d5-798x5            1/1     Running     0               4m2s    10.104.18.134   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querycoord-7bdc749f56-z9f6c      1/1     Running     0               4m2s    10.104.12.37    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-698d64dd64-5gl2m       1/1     Running     0               4m2s    10.104.18.135   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-698d64dd64-7h9tq       1/1     Running     0               4m2s    10.104.4.18     4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-698d64dd64-z6hzb       1/1     Running     0               4m2s    10.104.12.41    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-rootcoord-77d79f5776-wjtjd       1/1     Running     0               4m2s    10.104.12.38    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-0                                 1/1     Running     0               4m2s    10.104.18.136   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-1                                 1/1     Running     0               4m2s    10.104.14.145   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-2                                 1/1     Running     0               4m2s    10.104.19.120   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-3                                 1/1     Running     0               4m2s    10.104.5.111    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-0                         1/1     Running     0               4m2s    10.104.6.110    4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-1                         1/1     Running     0               4m2s    10.104.13.23    4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-2                         1/1     Running     0               4m1s    10.104.19.122   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-init-4hn6s                0/1     Completed   0               4m2s    10.104.18.128   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-broker-0                         1/1     Running     0               4m2s    10.104.5.108    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-proxy-0                          1/1     Running     0               4m2s    10.104.14.142   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-pulsar-init-9sg5k                0/1     Completed   0               4m2s    10.104.18.129   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-recovery-0                       1/1     Running     0               4m2s    10.104.12.40    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-0                      1/1     Running     0               4m2s    10.104.12.43    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-1                      1/1     Running     0               3m21s   10.104.13.25    4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-2                      1/1     Running     0               2m45s   10.104.6.114    4am-node13   <none>           <none>

NAME                                                              READY   STATUS                   RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-ivfsq8-scene-high-etcd-0                                  1/1     Running                  0                19h     10.104.19.117   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-1                                  1/1     Running                  0                19h     10.104.14.146   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-2                                  1/1     Running                  0                19h     10.104.5.113    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datacoord-649884d8b9-bpqlg       1/1     Running                  16 (6h4m ago)    19h     10.104.18.131   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-59fb787b47-9vdlh        0/1     Running                  49 (3m16s ago)   19h     10.104.12.36    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-59fb787b47-l2kzp        1/1     Running                  49 (7m23s ago)   19h     10.104.1.82     4am-node10   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexcoord-7c79599795-nsghm      1/1     Running                  0                19h     10.104.18.132   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-58b8888c45-dwr44       1/1     Running                  0                19h     10.104.5.109    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-58b8888c45-fngbs       1/1     Running                  0                19h     10.104.18.133   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-58b8888c45-n57rr       1/1     Running                  0                19h     10.104.12.39    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-proxy-ff4bcc4d5-798x5            1/1     Running                  2 (10h ago)      19h     10.104.18.134   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querycoord-7bdc749f56-z9f6c      1/1     Running                  0                19h     10.104.12.37    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-698d64dd64-5gl2m       1/1     Running                  40 (5h34m ago)   19h     10.104.18.135   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-698d64dd64-7h9tq       1/1     Running                  27 (5h28m ago)   19h     10.104.4.18     4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-698d64dd64-z6hzb       1/1     Running                  25 (6h1m ago)    19h     10.104.12.41    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-rootcoord-77d79f5776-wjtjd       1/1     Running                  0                19h     10.104.12.38    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-0                                 1/1     Running                  0                19h     10.104.18.136   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-1                                 1/1     Running                  0                19h     10.104.14.145   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-2                                 1/1     Running                  0                19h     10.104.19.120   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-3                                 1/1     Running                  0                19h     10.104.5.111    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-0                         1/1     Running                  58 (10m ago)     19h     10.104.6.110    4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-1                         1/1     Running                  54 (10m ago)     19h     10.104.13.23    4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-2                         1/1     Running                  54 (9m24s ago)   19h     10.104.19.122   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-init-4hn6s                0/1     Completed                0                19h     10.104.18.128   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-broker-0                         1/1     Running                  3 (10h ago)      19h     10.104.5.108    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-proxy-0                          1/1     Running                  0                19h     10.104.14.142   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-pulsar-init-9sg5k                0/1     Completed                0                19h     10.104.18.129   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-recovery-0                       1/1     Running                  0                19h     10.104.12.40    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-0                      1/1     Running                  0                19h     10.104.12.43    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-1                      1/1     Running                  0                19h     10.104.13.25    4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-2                      1/1     Running                  0                19h     10.104.6.114    4am-node13   <none>           <none>

exit_code.csv

New recurring scene

server argo task: fouramf-gm62x

deploy_config: fouramf-server-lb-2qn-2dn-large
case_params: --milvus_tag=master-20230803-68ecf49e -k test_server_custom_parameters -s --deploy_mode=cluster --deploy_retain --release_name=lb-helm-hnsw-scene-high

client argo task: fouramf-concurrent-zz4lx

other_params: --deploy_skip -s --release_name=lb-helm-hnsw-scene-high --case_skip_prepare_clean --case_skip_clean_collection
case_names: [ {"id": 1, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-hnsw-replica2-shard2-search-query-scene-high"}, {"id": 2, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-hnsw-replica1-shard2-search-query-scene-high"}, {"id": 3, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-hnsw-replica2-shard2-search-query-scene-high"}, {"id": 4, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-hnsw-replica1-shard2-search-query-scene-high"}, {"id": 5, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-hnsw-replica2-shard2-search-query-scene-high"}, {"id": 6, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-hnsw-replica1-shard2-search-query-scene-high"}, {"id": 7, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-hnsw-replica2-shard2-search-query-scene-high"}, {"id": 8, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-hnsw-replica1-shard2-search-query-scene-high"}, {"id": 9, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-hnsw-replica2-shard2-search-query-scene-high"}, {"id": 10, "case_name": "-k test_concurrent_locust_custom_parameters", "deploy_config": "empty-file", "case_params": "fouramf-client-sift-hnsw-replica1-shard2-search-query-scene-high"} ]
test-sdk-version: 2.4.0.dev73

HNSW index，high concurrent request；
During the request process, proxy report error query node is unreachable, resulting in request failure. The specific client-side error is as follows clients.log

[2023-08-03 08:16:35,869 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-08-03 08:16:34.699245', 'RPC error': '2023-08-03 08:16:35.869013'}> (decorators.py:108)
[2023-08-03 08:16:35,869 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)> (api_request.py:53)
[2023-08-03 08:16:35,869 - ERROR - fouram]: [CheckFunc] query request check failed, response:<MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)> (func_check.py:46)
[2023-08-03 08:16:36,065 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to search on all shard leaders)>, <Time:{'RPC start': '2023-08-03 08:16:35.827265', 'RPC error': '2023-08-03 08:16:36.065795'}> (decorators.py:108)
[2023-08-03 08:16:36,066 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to search on all shard leaders)> (api_request.py:53)
[2023-08-03 08:16:36,066 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to search on all shard leaders)> (func_check.py:46)
[2023-08-03 08:16:40,136 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)>, <Time:{'RPC start': '2023-08-03 08:16:39.931190', 'RPC error': '2023-08-03 08:16:40.136045'}> (decorators.py:108)
[2023-08-03 08:16:40,136 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)> (api_request.py:53)
[2023-08-03 08:16:40,136 - ERROR - fouram]: [CheckFunc] query request check failed, response:<MilvusException: (code=1, message=attempt #0: all available nodes are unreachable: service unavailable: fail to query on all shard leaders)> (func_check.py:46)
[2023-08-03 10:20:11,840 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443297777073850835 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 10:20:11.835557', 'RPC error': '2023-08-03 10:20:11.840451'}> (decorators.py:108)
[2023-08-03 10:20:11,841 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443297777073850835 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 09:27:26.155313', 'RPC error': '2023-08-03 10:20:11.841463'}> (decorators.py:108)
[2023-08-03 10:20:11,841 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443297777073850835 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 09:25:10.944856', 'RPC error': '2023-08-03 10:20:11.841606'}> (decorators.py:108)
[2023-08-03 10:20:11,842 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=collection 443297777073850835 has not been loaded to memory or load failed)> (api_request.py:53)
[2023-08-03 10:20:11,842 - ERROR - fouram]: [CheckFunc] load request check failed, response:<MilvusException: (code=1, message=collection 443297777073850835 has not been loaded to memory or load failed)> (func_check.py:52)
[2023-08-03 10:20:11,842 - ERROR - fouram]: [func_time_catch] :  (api_request.py:120)
[2023-08-03 10:20:11,861 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 443297777073431147 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 10:20:11.850235', 'RPC error': '2023-08-03 10:20:11.861194'}> (decorators.py:108)
[2023-08-03 10:20:11,861 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 443297777073431147 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 09:32:17.061831', 'RPC error': '2023-08-03 10:20:11.861527'}> (decorators.py:108)
[2023-08-03 10:20:11,861 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 443297777073431147 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-08-03 09:25:20.047238', 'RPC error': '2023-08-03 10:20:11.861712'}> (decorators.py:108)
[2023-08-03 10:20:11,862 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=collection 443297777073431147 has not been loaded to memory or load failed)> (api_request.py:53)

server:

NAME                                                              READY   STATUS                   RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-hnsw-scene-high-etcd-0                                    1/1     Running                  0                19h     10.104.13.30    4am-node16   <none>           <none>
lb-helm-hnsw-scene-high-etcd-1                                    1/1     Running                  0                19h     10.104.19.132   4am-node28   <none>           <none>
lb-helm-hnsw-scene-high-etcd-2                                    1/1     Running                  0                19h     10.104.21.254   4am-node24   <none>           <none>
lb-helm-hnsw-scene-high-milvus-datacoord-7b78599c7f-4z4fp         1/1     Running                  0                19h     10.104.21.249   4am-node24   <none>           <none>
lb-helm-hnsw-scene-high-milvus-datanode-6bfc765d76-wp4vm          1/1     Running                  0                19h     10.104.14.151   4am-node18   <none>           <none>
lb-helm-hnsw-scene-high-milvus-datanode-6bfc765d76-zp9kh          1/1     Running                  0                19h     10.104.19.126   4am-node28   <none>           <none>
lb-helm-hnsw-scene-high-milvus-indexcoord-59869cb5bd-h8wv5        1/1     Running                  0                19h     10.104.19.124   4am-node28   <none>           <none>
lb-helm-hnsw-scene-high-milvus-indexnode-c56cb48dd-htlfk          1/1     Running                  0                19h     10.104.1.83     4am-node10   <none>           <none>
lb-helm-hnsw-scene-high-milvus-indexnode-c56cb48dd-nfgx5          1/1     Running                  0                19h     10.104.6.115    4am-node13   <none>           <none>
lb-helm-hnsw-scene-high-milvus-proxy-6565745d7b-7jmqz             1/1     Running                  0                19h     10.104.19.123   4am-node28   <none>           <none>
lb-helm-hnsw-scene-high-milvus-querycoord-5b9ff84695-b7qdn        1/1     Running                  0                19h     10.104.14.149   4am-node18   <none>           <none>
lb-helm-hnsw-scene-high-milvus-querynode-75b87774fc-679t4         1/1     Running                  0                19h     10.104.13.26    4am-node16   <none>           <none>
lb-helm-hnsw-scene-high-milvus-querynode-75b87774fc-b6fjr         1/1     Running                  0                19h     10.104.5.116    4am-node12   <none>           <none>
lb-helm-hnsw-scene-high-milvus-rootcoord-cfc78f479-b2crc          1/1     Running                  0                19h     10.104.21.248   4am-node24   <none>           <none>
lb-helm-hnsw-scene-high-minio-0                                   1/1     Running                  0                19h     10.104.5.120    4am-node12   <none>           <none>
lb-helm-hnsw-scene-high-minio-1                                   1/1     Running                  0                19h     10.104.19.130   4am-node28   <none>           <none>
lb-helm-hnsw-scene-high-minio-2                                   1/1     Running                  0                19h     10.104.21.6     4am-node24   <none>           <none>
lb-helm-hnsw-scene-high-minio-3                                   1/1     Running                  0                19h     10.104.9.64     4am-node14   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-bookie-0                           1/1     Running                  0                19h     10.104.14.155   4am-node18   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-bookie-1                           1/1     Running                  0                19h     10.104.21.7     4am-node24   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-bookie-2                           1/1     Running                  0                19h     10.104.9.66     4am-node14   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-bookie-init-6q9m7                  0/1     Completed                0                19h     10.104.19.125   4am-node28   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-broker-0                           1/1     Running                  0                19h     10.104.19.128   4am-node28   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-proxy-0                            1/1     Running                  0                19h     10.104.14.152   4am-node18   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-pulsar-init-qp8zt                  0/1     Completed                0                19h     10.104.19.127   4am-node28   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-recovery-0                         1/1     Running                  0                19h     10.104.13.27    4am-node16   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-zookeeper-0                        1/1     Running                  0                19h     10.104.13.31    4am-node16   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-zookeeper-1                        1/1     Running                  0                19h     10.104.14.157   4am-node18   <none>           <none>
lb-helm-hnsw-scene-high-pulsar-zookeeper-2                        1/1     Running                  0                19h     10.104.17.147   4am-node23   <none>           <none>

/assign @wangting0128 pls check with #26135

Recurrent

image: master-20230807-07f08daf argo task: fouramf-server-client-concurrent-6dlxz

server:

NAME                                                              READY   STATUS      RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-ivfsq8-scene-high-etcd-0                                  1/1     Running     0               18h     10.104.17.28    4am-node23   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-1                                  1/1     Running     0               18h     10.104.21.85    4am-node24   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-2                                  1/1     Running     0               18h     10.104.19.116   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datacoord-8599b4bf67-qg7t8       1/1     Running     0               18h     10.104.6.125    4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-844dd599b5-lggg9        1/1     Running     1 (18h ago)     18h     10.104.6.130    4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-844dd599b5-qcggj        1/1     Running     0               18h     10.104.9.226    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexcoord-7fb6758985-fvddx      1/1     Running     0               18h     10.104.6.127    4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-5cd7d4f7f-576sj        1/1     Running     0               18h     10.104.17.25    4am-node23   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-5cd7d4f7f-mk4qb        1/1     Running     0               18h     10.104.15.90    4am-node20   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-5cd7d4f7f-nk8tv        1/1     Running     0               18h     10.104.23.176   4am-node27   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-proxy-b4744c647-ll765            1/1     Running     1 (18h ago)     18h     10.104.19.110   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querycoord-689898f8b6-sh4z8      1/1     Running     1 (18h ago)     18h     10.104.6.126    4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-55b76cbcfd-9n6dx       1/1     Running     0               18h     10.104.9.227    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-55b76cbcfd-jh7l7       1/1     Running     0               18h     10.104.6.129    4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-55b76cbcfd-lqnhv       1/1     Running     0               18h     10.104.5.35     4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-rootcoord-7b574455f8-dbngg       1/1     Running     0               18h     10.104.6.128    4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-0                                 1/1     Running     0               18h     10.104.17.29    4am-node23   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-1                                 1/1     Running     0               18h     10.104.14.143   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-2                                 1/1     Running     0               18h     10.104.21.87    4am-node24   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-3                                 1/1     Running     0               18h     10.104.12.109   4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-0                         1/1     Running     0               18h     10.104.24.76    4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-1                         1/1     Running     0               18h     10.104.19.115   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-2                         1/1     Running     0               18h     10.104.17.32    4am-node23   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-init-xrwwg                0/1     Completed   0               18h     10.104.21.81    4am-node24   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-broker-0                         1/1     Running     0               18h     10.104.12.107   4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-proxy-0                          1/1     Running     0               18h     10.104.19.111   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-pulsar-init-7jkz2                0/1     Completed   0               18h     10.104.21.80    4am-node24   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-recovery-0                       1/1     Running     0               18h     10.104.14.141   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-0                      1/1     Running     0               18h     10.104.21.83    4am-node24   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-1                      1/1     Running     0               18h     10.104.14.145   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-2                      1/1     Running     0               18h     10.104.17.38    4am-node23   <none>           <none>

logs:

client log: clients.log

Recurrent

image tag: master-20230809-12a7747a argo task: fouramf-server-client-concurrent-rhmgk

server:

NAME                                                              READY   STATUS      RESTARTS       AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-ivfsq8-scene-high-etcd-0                                  1/1     Running     1 (144m ago)   145m    10.104.5.175    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-1                                  1/1     Running     1 (144m ago)   145m    10.104.20.198   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-2                                  1/1     Running     0              145m    10.104.14.183   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datacoord-6d46cc77fc-hj2jk       1/1     Running     0              145m    10.104.14.173   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95        1/1     Running     1 (141m ago)   145m    10.104.9.203    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-vvz8f        1/1     Running     1 (141m ago)   145m    10.104.20.193   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexcoord-5fc6f9f47-fsqmn       1/1     Running     0              145m    10.104.20.194   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-857bf66f49-jq7p5       1/1     Running     0              145m    10.104.24.48    4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-857bf66f49-s2tqv       1/1     Running     0              145m    10.104.4.241    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-857bf66f49-vfjk9       1/1     Running     0              145m    10.104.6.242    4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9           1/1     Running     1 (141m ago)   145m    10.104.9.205    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querycoord-5bbdfb645b-2v7h4      1/1     Running     1 (141m ago)   145m    10.104.5.165    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-9qzw6        1/1     Running     0              145m    10.104.9.206    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-ddp6c        1/1     Running     0              145m    10.104.18.179   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-pm6xz        1/1     Running     0              145m    10.104.15.115   4am-node20   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-rootcoord-85fcf85d56-dg7xr       1/1     Running     1 (141m ago)   145m    10.104.24.47    4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-0                                 1/1     Running     0              145m    10.104.4.243    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-1                                 1/1     Running     0              145m    10.104.20.196   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-2                                 1/1     Running     0              145m    10.104.5.176    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-3                                 1/1     Running     0              145m    10.104.14.182   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-0                         1/1     Running     0              145m    10.104.5.168    4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-1                         1/1     Running     0              145m    10.104.14.184   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-2                         1/1     Running     0              145m    10.104.20.205   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-init-qlwpx                0/1     Completed   0              145m    10.104.9.204    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-broker-0                         1/1     Running     0              145m    10.104.4.240    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-proxy-0                          1/1     Running     0              145m    10.104.9.202    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-pulsar-init-dvsfp                0/1     Completed   0              145m    10.104.9.201    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-recovery-0                       1/1     Running     0              145m    10.104.24.49    4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-0                      1/1     Running     0              145m    10.104.20.202   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-1                      1/1     Running     0              145m    10.104.12.10    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-2                      1/1     Running     0              143m    10.104.1.175    4am-node10   <none>           <none>

logs:

querynode
querynode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-9qzw6
[2023/08/09 05:10:03.752 +00:00] [ERROR] [funcutil/parallel.go:88] [loadRemoteFunc] [error="failed to read file/stats_log/443430610504212206/443430610504212207/443430610589416302/100/1: Get \"http://lb-helm-ivfsq8-scene-high-minio:9000/milvus-bucket/file/stats_log/443430610504212206/443430610504212207/443430610589416302/100/1\": context canceled"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:88"]
Show context
querynode
querynode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-9qzw6
[2023/08/09 05:04:01.753 +00:00] [ERROR] [funcutil/parallel.go:88] [loadRemoteFunc] [error="failed to read file/stats_log/443430610503812128/443430610503812129/443430610562614957/100/1: Get \"http://lb-helm-ivfsq8-scene-high-minio:9000/milvus-bucket/file/stats_log/443430610503812128/443430610503812129/443430610562614957/100/1\": context canceled"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:88"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:59:49.048 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=20] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:356 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:59:19.015 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=10] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:356 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:58:57.984 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=0] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:356 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:58:57.976 +00:00] [ERROR] [config/manager.go:250] ["Get configuration by items failed"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*Manager).pullSourceConfigs\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:250\ngithub.com/milvus-io/milvus/pkg/config.(*Manager).AddSource\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:187\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).initConfigsFromRemote\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:160\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:115\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*hookConfig).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/hook_config.go:20\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:121\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:97\ngithub.com/milvus-io/milvus/pkg/util/paramtable.Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/runtime.go:29\ngithub.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:268\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:58:43.815 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=30] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:356 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:58:13.728 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=20] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:356 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:57:43.693 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=10] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:356 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-9qzw6
[2023/08/09 03:57:41.366 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=20] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:147\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:247\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:123\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:208\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-ddp6c
[2023/08/09 03:57:40.439 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=20] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:147\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:247\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:123\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:208\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node20
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-pm6xz
[2023/08/09 03:57:39.188 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=20] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:147\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:247\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:123\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:208\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:57:22.663 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=0] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:356 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:515\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:56:21.414 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:499\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-9qzw6
[2023/08/09 03:56:21.353 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:147\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:247\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:123\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:208\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-ddp6c
[2023/08/09 03:56:20.426 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:147\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:247\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:123\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:208\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node20
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-pm6xz
[2023/08/09 03:56:19.175 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:147\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:247\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:123\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:208\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:55:10.403 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:499\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:339\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-9qzw6
[2023/08/09 03:55:10.338 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:147\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:247\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:123\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:208\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-ddp6c
[2023/08/09 03:55:09.413 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:147\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:247\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:123\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:208\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node20
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-pm6xz
[2023/08/09 03:55:08.163 +00:00] [ERROR] [sessionutil/session_util.go:232] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:232\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:147\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:247\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/querynodev2/server.go:244\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:123\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querynode/service.go:208\ngithub.com/milvus-io/milvus/cmd/components.(*QueryNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_node.go:54\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:55:05.400 +00:00] [ERROR] [config/manager.go:250] ["Get configuration by items failed"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*Manager).pullSourceConfigs\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:250\ngithub.com/milvus-io/milvus/pkg/config.(*Manager).AddSource\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:187\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).initConfigsFromRemote\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:160\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:115\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*hookConfig).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/hook_config.go:20\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:121\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:97\ngithub.com/milvus-io/milvus/pkg/util/paramtable.Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/runtime.go:29\ngithub.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:268\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
querynode
querynode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-9qzw6
[2023/08/09 03:55:05.234 +00:00] [ERROR] [config/manager.go:250] ["Get configuration by items failed"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*Manager).pullSourceConfigs\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:250\ngithub.com/milvus-io/milvus/pkg/config.(*Manager).AddSource\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:187\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).initConfigsFromRemote\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:160\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:115\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*hookConfig).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/hook_config.go:20\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:121\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:97\ngithub.com/milvus-io/milvus/pkg/util/paramtable.Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/runtime.go:29\ngithub.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:268\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
querynode
querynode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-ddp6c
[2023/08/09 03:55:04.310 +00:00] [ERROR] [config/manager.go:250] ["Get configuration by items failed"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*Manager).pullSourceConfigs\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:250\ngithub.com/milvus-io/milvus/pkg/config.(*Manager).AddSource\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:187\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).initConfigsFromRemote\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:160\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:115\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*hookConfig).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/hook_config.go:20\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:121\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:97\ngithub.com/milvus-io/milvus/pkg/util/paramtable.Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/runtime.go:29\ngithub.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:268\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
querynode
querynode
4am-node20
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-pm6xz
[2023/08/09 03:55:03.060 +00:00] [ERROR] [config/manager.go:250] ["Get configuration by items failed"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*Manager).pullSourceConfigs\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:250\ngithub.com/milvus-io/milvus/pkg/config.(*Manager).AddSource\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:187\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).initConfigsFromRemote\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:160\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:115\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*hookConfig).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/hook_config.go:20\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:121\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:97\ngithub.com/milvus-io/milvus/pkg/util/paramtable.Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/runtime.go:29\ngithub.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:268\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
proxy
proxy
4am-node14
lb-helm-ivfsq8-scene-high-milvus-proxy-54d95d79f7-8mwf9
[2023/08/09 03:55:02.394 +00:00] [ERROR] [config/manager.go:250] ["Get configuration by items failed"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*Manager).pullSourceConfigs\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:250\ngithub.com/milvus-io/milvus/pkg/config.(*Manager).AddSource\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:187\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).initConfigsFromRemote\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:160\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:115\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ServiceParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/service_param.go:60\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:105\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:97\ngithub.com/milvus-io/milvus/pkg/util/paramtable.Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/runtime.go:29\ngithub.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:268\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
querynode
querynode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-9qzw6
[2023/08/09 03:55:02.228 +00:00] [ERROR] [config/manager.go:250] ["Get configuration by items failed"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*Manager).pullSourceConfigs\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:250\ngithub.com/milvus-io/milvus/pkg/config.(*Manager).AddSource\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:187\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).initConfigsFromRemote\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:160\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:115\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ServiceParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/service_param.go:60\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:105\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:97\ngithub.com/milvus-io/milvus/pkg/util/paramtable.Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/runtime.go:29\ngithub.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:268\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
querynode
querynode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-ddp6c
[2023/08/09 03:55:01.304 +00:00] [ERROR] [config/manager.go:250] ["Get configuration by items failed"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*Manager).pullSourceConfigs\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:250\ngithub.com/milvus-io/milvus/pkg/config.(*Manager).AddSource\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:187\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).initConfigsFromRemote\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:160\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:115\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ServiceParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/service_param.go:60\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:105\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:97\ngithub.com/milvus-io/milvus/pkg/util/paramtable.Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/runtime.go:29\ngithub.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:268\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]
querynode
querynode
4am-node20
lb-helm-ivfsq8-scene-high-milvus-querynode-cf6f5bd9b-pm6xz
[2023/08/09 03:55:00.055 +00:00] [ERROR] [config/manager.go:250] ["Get configuration by items failed"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/config.(*Manager).pullSourceConfigs\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:250\ngithub.com/milvus-io/milvus/pkg/config.(*Manager).AddSource\n\t/go/src/github.com/milvus-io/milvus/pkg/config/manager.go:187\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).initConfigsFromRemote\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:160\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*BaseTable).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/base_table.go:115\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ServiceParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/service_param.go:60\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:105\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:98\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/pkg/util/paramtable.(*ComponentParam).Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/component_param.go:97\ngithub.com/milvus-io/milvus/pkg/util/paramtable.Init\n\t/go/src/github.com/milvus-io/milvus/pkg/util/paramtable/runtime.go:29\ngithub.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:268\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"]

clients log:

Recurrent

image tag: master-20230809-889424b3 argo task: fouramf-server-client-concurrent-9fsrw

server:

NAME                                                              READY   STATUS      RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-ivfsq8-scene-high-etcd-0                                  1/1     Running     0                15h     10.104.18.241   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-1                                  1/1     Running     0                15h     10.104.9.3      4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-2                                  1/1     Running     0                15h     10.104.12.32    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datacoord-7754c98d77-hdcqd       1/1     Running     94 (154m ago)    15h     10.104.18.236   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-dd5f8f579-hdj5q         1/1     Running     108 (105s ago)   15h     10.104.9.250    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-dd5f8f579-mfqsk         1/1     Running     15 (26m ago)     15h     10.104.18.232   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexcoord-6c7bf79787-b7gf6      1/1     Running     0                15h     10.104.18.235   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-fdcd547f7-5xrqx        1/1     Running     0                15h     10.104.15.133   4am-node20   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-fdcd547f7-jgrtj        1/1     Running     0                15h     10.104.9.249    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-fdcd547f7-t85ks        1/1     Running     0                15h     10.104.18.229   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-proxy-5866845d8f-w762x           1/1     Running     0                15h     10.104.18.228   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querycoord-bb74f5b8d-c6l4f       1/1     Running     0                15h     10.104.9.248    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-6c8467f444-bpv29       1/1     Running     0                15h     10.104.15.132   4am-node20   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-6c8467f444-bqc46       1/1     Running     0                15h     10.104.18.238   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-6c8467f444-dt4fv       1/1     Running     0                15h     10.104.9.252    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-rootcoord-c745b78f6-ltv86        1/1     Running     0                15h     10.104.18.227   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-0                                 1/1     Running     0                15h     10.104.18.243   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-1                                 1/1     Running     0                15h     10.104.9.8      4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-2                                 1/1     Running     0                15h     10.104.12.34    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-3                                 1/1     Running     0                15h     10.104.4.101    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-0                         0/1     Running     4 (4s ago)       15h     10.104.9.9      4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-1                         1/1     Running     3 (6m35s ago)    15h     10.104.12.35    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-2                         1/1     Running     1 (21m ago)      15h     10.104.4.102    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-init-srn6b                0/1     Completed   0                15h     10.104.18.231   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-broker-0                         1/1     Running     0                15h     10.104.18.234   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-proxy-0                          1/1     Running     0                15h     10.104.9.251    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-pulsar-init-x9kdr                0/1     Completed   0                15h     10.104.18.230   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-recovery-0                       1/1     Running     0                15h     10.104.18.233   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-0                      1/1     Running     0                15h     10.104.18.242   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-1                      1/1     Running     0                15h     10.104.12.37    4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-2                      1/1     Running     0                15h     10.104.4.104    4am-node11   <none>           <none>

dataNode and dataCoord panic:

datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-dd5f8f579-hdj5q
stderr
    /usr/local/go/src/runtime/panic.go:939 +0x389 fp=0xc00449f970 sp=0xc00449f8b0 pc=0x152d369
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-dd5f8f579-hdj5q
stderr
panic({0x3bbee20, 0xc0037258c0})
Show context
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-dd5f8f579-hdj5q
stderr
    /usr/local/go/src/runtime/panic.go:535 +0x7f fp=0xc00449f8b0 sp=0xc00449f850 pc=0x152c67f
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-dd5f8f579-hdj5q
stderr
runtime.preprintpanics(0x46?)
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-dd5f8f579-hdj5q
stderr
    /usr/local/go/src/runtime/panic.go:941 +0x397 fp=0xc00471f970 sp=0xc00471f8b0 pc=0x152d377
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-dd5f8f579-hdj5q
stderr
panic({0x3bbee20, 0xc0037c20a8})
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-dd5f8f579-hdj5q
stderr
panic: attempt #0: err: failed to connect 10.104.18.236:13333, reason: connection error: desc = "transport: error while dialing: dial tcp 10.104.18.236:13333: connect: connection refused"
datacoord
datacoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datacoord-7754c98d77-hdcqd
stdout
[2023/08/09 14:47:54.377 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:378\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:404\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:883\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:777\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:656\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:740"]
datacoord
datacoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datacoord-7754c98d77-hdcqd
stdout
[2023/08/09 14:13:14.604 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:378\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:404\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:883\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:777\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:656\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:740"]
datacoord
datacoord
4am-node18
lb-helm-ivfsq8-scene-high-milvus-datacoord-6d46cc77fc-hj2jk
stdout
[2023/08/09 11:28:58.620 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:378\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:404\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:883\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:777\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:656\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:740"]
datacoord
datacoord
4am-node18
lb-helm-ivfsq8-scene-high-milvus-datacoord-6d46cc77fc-hj2jk
stdout
[2023/08/09 10:34:14.103 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:378\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:404\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:883\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:777\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:656\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:740"]
datacoord
datacoord
4am-node18
lb-helm-ivfsq8-scene-high-milvus-datacoord-6d46cc77fc-hj2jk
stdout
[2023/08/09 09:57:07.659 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:378\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:404\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:883\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:777\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:656\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:740"]
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-vvz8f
stderr
    /usr/local/go/src/runtime/panic.go:939 +0x389 fp=0xc0016ef970 sp=0xc0016ef8b0 pc=0x1536369
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-vvz8f
stderr
panic({0x3bc7dc0, 0xc00304bfe0})
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-vvz8f
stderr
    /usr/local/go/src/runtime/panic.go:535 +0x7f fp=0xc0016ef8b0 sp=0xc0016ef850 pc=0x153567f
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-vvz8f
stderr
runtime.preprintpanics(0x46?)
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-vvz8f
stderr
    /usr/local/go/src/runtime/panic.go:941 +0x397 fp=0xc00150f970 sp=0xc00150f8b0 pc=0x1536377
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-vvz8f
stderr
panic({0x3bc7dc0, 0xc003801f50})
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-vvz8f
stderr
panic: attempt #0: err: failed to connect 10.104.14.173:13333, reason: connection error: desc = "transport: error while dialing: dial tcp 10.104.14.173:13333: connect: connection refused"
datacoord
datacoord
4am-node18
lb-helm-ivfsq8-scene-high-milvus-datacoord-6d46cc77fc-hj2jk
stdout
[2023/08/09 09:24:27.263 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:378\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:404\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:883\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:777\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:656\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:740"]
datacoord
datacoord
4am-node18
lb-helm-ivfsq8-scene-high-milvus-datacoord-6d46cc77fc-hj2jk
stdout
[2023/08/09 08:52:34.621 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:378\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:404\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:883\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:777\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:656\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:740"]
datacoord
datacoord
4am-node18
lb-helm-ivfsq8-scene-high-milvus-datacoord-6d46cc77fc-hj2jk
stdout
[2023/08/09 08:20:11.080 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:378\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:404\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:883\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:777\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:656\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:740"]
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95
stderr
    /usr/local/go/src/runtime/panic.go:941 +0x397 fp=0xc0038eb970 sp=0xc0038eb8b0 pc=0x1536377
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95
stderr
panic({0x3bc7dc0, 0xc00199c9a8})
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95
stderr
panic: attempt #0: err: failed to connect 10.104.14.173:13333, reason: connection error: desc = "transport: error while dialing: dial tcp 10.104.14.173:13333: connect: connection refused"
datacoord
datacoord
4am-node18
lb-helm-ivfsq8-scene-high-milvus-datacoord-6d46cc77fc-hj2jk
stdout
[2023/08/09 07:47:36.935 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:378\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:404\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:883\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:777\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:656\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:740"]
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95
stderr
    /usr/local/go/src/runtime/panic.go:939 +0x389 fp=0xc00722bb50 sp=0xc00722ba90 pc=0x1536369
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95
stderr
panic({0x3bc7dc0, 0xc009f8e8a0})
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95
stderr
    /usr/local/go/src/runtime/panic.go:535 +0x7f fp=0xc00722ba90 sp=0xc00722ba30 pc=0x153567f
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95
stderr
runtime.preprintpanics(0xc00eb98fa0?)
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95
stderr
    /usr/local/go/src/runtime/panic.go:941 +0x397 fp=0xc002f87b50 sp=0xc002f87a90 pc=0x1536377
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95
stderr
panic({0x3bc7dc0, 0xc009f09f98})
datanode
datanode
4am-node14
lb-helm-ivfsq8-scene-high-milvus-datanode-6c6db488bc-pdj95
stderr
panic: attempt #0: err: failed to connect 10.104.14.173:13333, reason: connection error: desc = "transport: error while dialing: dial tcp 10.104.14.173:13333: connect: connection refused"
datacoord
datacoord
4am-node18
lb-helm-ivfsq8-scene-high-milvus-datacoord-6d46cc77fc-hj2jk
stdout
[2023/08/09 07:12:38.907 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:378\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:404\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:883\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:777\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:656\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:740"]

dataNode and dataCoord panic:

clients log:

server config:

{
     "queryNode": {
          "resources": {
               "limits": {
                    "cpu": "50.0",
                    "memory": "100Gi"
               },
               "requests": {
                    "cpu": "25.0",
                    "memory": "50Gi"
               }
          },
          "replicas": 3
     },
     "indexNode": {
          "resources": {
               "limits": {
                    "cpu": "16.0",
                    "memory": "16Gi"
               },
               "requests": {
                    "cpu": "5.0",
                    "memory": "5Gi"
               }
          },
          "replicas": 3
     },
     "dataNode": {
          "resources": {
               "limits": {
                    "cpu": "2.0",
                    "memory": "16Gi"
               },
               "requests": {
                    "cpu": "2.0",
                    "memory": "2Gi"
               }
          },
          "replicas": 2
     },
     "cluster": {
          "enabled": true
     },
     "pulsar": {
          "bookkeeper": {
               "volumes": {
                    "journal": {
                         "storageClassName": "local-path"
                    },
                    "ledgers": {
                         "storageClassName": "local-path"
                    }
               }
          },
          "zookeeper": {
               "volumes": {
                    "data": {
                         "storageClassName": "local-path"
                    }
               }
          }
     },
     "kafka": {
          "persistence": {
               "storageClass": "local-path"
          }
     },
     "minio": {
          "persistence": {
               "storageClass": "local-path"
          },
          "metrics": {
               "podMonitor": {
                    "enabled": true
               }
          }
     },
     "etcd": {
          "global": {
               "storageClass": "local-path"
          },
          "metrics": {
               "enabled": true,
               "podMonitor": {
                    "enabled": true
               }
          }
     },
     "metrics": {
          "serviceMonitor": {
               "enabled": true
          }
     },
     "log": {
          "level": "debug"
     },
     "image": {
          "all": {
               "repository": "harbor.milvus.io/milvus/milvus",
               "tag": "master-20230809-889424b3"
          }
     }
}

/assign @SimFG please help on it

Recurrent

image tag: master-20230816-911406c8 argo task : fouramf-server-client-concurrent-xlrwl

server:

NAME                                                              READY   STATUS                   RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-ivfsq8-scene-high-etcd-0                                  1/1     Running                  0               5h28m   10.104.19.127   4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-1                                  1/1     Running                  0               5h28m   10.104.21.160   4am-node24   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-2                                  1/1     Running                  0               5h28m   10.104.18.153   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datacoord-548d45b675-c29rp       0/1     Running                  2 (2m17s ago)   5h28m   10.104.16.36    4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-56jz7        1/1     Running                  2 (39m ago)     5h28m   10.104.16.35    4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt        1/1     Running                  1 (80m ago)     5h28m   10.104.20.101   4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexcoord-7947477db5-g8dlt      1/1     Running                  0               5h28m   10.104.18.148   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-88c5d88c9-bckqr        1/1     Running                  0               5h28m   10.104.16.37    4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-88c5d88c9-pxf2s        1/1     Running                  0               5h28m   10.104.17.200   4am-node23   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-88c5d88c9-stmmw        1/1     Running                  0               5h28m   10.104.20.99    4am-node22   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-proxy-646bfbc9f9-tlm5h           1/1     Running                  0               5h28m   10.104.14.99    4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querycoord-77b8865898-5hchc      1/1     Running                  0               5h28m   10.104.16.34    4am-node21   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-54b6bbc55d-gjgmt       1/1     Running                  0               5h28m   10.104.24.106   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-54b6bbc55d-m9z4n       1/1     Running                  0               5h28m   10.104.13.238   4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-54b6bbc55d-pvzzs       1/1     Running                  0               5h28m   10.104.12.192   4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-rootcoord-64f6cb9bd6-c4g6q       1/1     Running                  0               5h28m   10.104.14.100   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-0                                 1/1     Running                  0               5h28m   10.104.1.173    4am-node10   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-1                                 1/1     Running                  0               5h28m   10.104.21.159   4am-node24   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-2                                 1/1     Running                  0               5h28m   10.104.18.152   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-3                                 1/1     Running                  0               5h28m   10.104.4.131    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-0                         1/1     Running                  0               5h28m   10.104.18.155   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-1                         1/1     Running                  0               5h28m   10.104.4.134    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-2                         1/1     Running                  0               5h28m   10.104.9.68     4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-init-ntqbj                0/1     Completed                0               5h28m   10.104.14.98    4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-broker-0                         1/1     Running                  0               5h28m   10.104.14.96    4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-proxy-0                          1/1     Running                  0               5h28m   10.104.5.89     4am-node12   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-pulsar-init-rcskg                0/1     Completed                0               5h28m   10.104.14.97    4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-recovery-0                       1/1     Running                  0               5h28m   10.104.14.101   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-0                      1/1     Running                  0               5h28m   10.104.12.191   4am-node17   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-1                      1/1     Running                  0               5h28m   10.104.18.157   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-2                      1/1     Running                  0               5h27m   10.104.16.40    4am-node21   <none>           <none>

panic:

datacoord
datacoord
4am-node21
lb-helm-ivfsq8-scene-high-milvus-datacoord-548d45b675-c29rp
stdout
[2023/08/16 10:47:25.846 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:381\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:407\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:884\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).Reassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:778\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:657\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:741"]
Show context
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt
stderr
panic: attempt #0: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #1: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #2: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #3: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #4: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #5: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #6: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #7: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #8: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #9: data service DropVirtualChannel failed, reason = DataCoord is not serving
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt
stderr
panic({0x3be2440, 0xc0352f4198})
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt
stderr
    /usr/local/go/src/runtime/panic.go:941 +0x397 fp=0xc020e09b50 sp=0xc020e09a90 pc=0x153a377
datanode
datanode
4am-node21
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-56jz7
stderr
panic: attempt #0: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #1: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #2: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #3: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #4: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #5: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #6: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #7: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #8: data service DropVirtualChannel failed, reason = DataCoord is not serving: attempt #9: data service DropVirtualChannel failed, reason = DataCoord is not serving
datanode
datanode
4am-node21
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-56jz7
stderr
panic({0x3be2440, 0xc029214000})
datanode
datanode
4am-node21
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-56jz7
stderr
    /usr/local/go/src/runtime/panic.go:941 +0x397 fp=0xc024025b50 sp=0xc024025a90 pc=0x153a377
datacoord
datacoord
4am-node21
lb-helm-ivfsq8-scene-high-milvus-datacoord-548d45b675-c29rp
stdout
[2023/08/16 12:06:34.189 +00:00] [FATAL] [logutil/logutil.go:132] [panic] [recover="\"datacoord ServerHandler HasCollection finally failed\""] [stack="github.com/milvus-io/milvus/pkg/util/logutil.LogPanic\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/logutil.go:132\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:838\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).HasCollection\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:381\ngithub.com/milvus-io/milvus/internal/datacoord.(*ServerHandler).CheckShouldDropChannel\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/handler.go:407\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).isMarkedDrop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:884\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).CleanupAndReassign\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:825\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).processAck\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:647\ngithub.com/milvus-io/milvus/internal/datacoord.(*ChannelManager).watchChannelStatesLoop\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/channel_manager.go:690"]
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt
stderr
panic: attempt #0: err: failed to connect 10.104.16.36:13333, reason: connection error: desc = "transport: error while dialing: dial tcp 10.104.16.36:13333: connect: connection refused"
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt
stderr
panic({0x3be2440, 0xc046cb30e0})
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt
stderr
    /usr/local/go/src/runtime/panic.go:941 +0x397 fp=0xc03cbff970 sp=0xc03cbff8b0 pc=0x153a377
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt
stderr
runtime.preprintpanics(0x46?)
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt
stderr
    /usr/local/go/src/runtime/panic.go:535 +0x7f fp=0xc0083b98b0 sp=0xc0083b9850 pc=0x153967f
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt
stderr
panic({0x3be2440, 0xc046cb3140})
datanode
datanode
4am-node22
lb-helm-ivfsq8-scene-high-milvus-datanode-56b6cfb7c7-c6hkt
stderr
    /usr/local/go/src/runtime/panic.go:939 +0x389 fp=0xc0083b9970 sp=0xc0083b98b0 pc=0x153a369

clients log:

The tickler may not stop normally, causing the program to hang, the memory keeps increasing, and finally restart. But it still fails after reboot because of wrong kv insertion. #https://github.com/milvus-io/milvus/pull/26470

Use the repaired pr to test and find new problems, collection load timed out, has been fixed @weiliu1031. #26435

Recurrent

image tag: master-20230818-74fb244b argo task: fouramf-server-client-concurrent-5bk4s

server:

NAME                                                              READY   STATUS                   RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-ivfsq8-scene-high-etcd-0                                  1/1     Running                  0               133m    10.104.4.250    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-1                                  1/1     Running                  0               133m    10.104.6.30     4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-2                                  1/1     Running                  0               133m    10.104.24.129   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datacoord-7d6b659d74-hfk7c       1/1     Running                  0               133m    10.104.18.108   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg         1/1     Running                  1 (129m ago)    133m    10.104.18.109   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp         1/1     Running                  0               133m    10.104.24.126   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexcoord-5cbbb47d88-rrtdp      1/1     Running                  0               133m    10.104.18.111   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-brkrr        1/1     Running                  0               133m    10.104.14.253   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-kh8tx        1/1     Running                  0               133m    10.104.13.239   4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-rgw9v        1/1     Running                  0               133m    10.104.18.113   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9           1/1     Running                  1 (129m ago)    133m    10.104.24.127   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd      1/1     Running                  1 (129m ago)    133m    10.104.18.107   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-5cbcc747d9-bw8lp       1/1     Running                  0               133m    10.104.24.173   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-5cbcc747d9-gw4xk       1/1     Running                  0               133m    10.104.13.244   4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-5cbcc747d9-q9z9z       1/1     Running                  0               133m    10.104.19.48    4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-rootcoord-784b96cbdc-wwvjb       1/1     Running                  0               133m    10.104.18.110   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-0                                 1/1     Running                  0               133m    10.104.18.116   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-1                                 1/1     Running                  0               133m    10.104.4.2      4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-2                                 1/1     Running                  0               133m    10.104.9.128    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-3                                 1/1     Running                  0               133m    10.104.24.131   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-0                         1/1     Running                  0               133m    10.104.18.118   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-1                         1/1     Running                  0               133m    10.104.4.6      4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-2                         1/1     Running                  0               133m    10.104.1.230    4am-node10   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-init-trr5k                0/1     Completed                0               133m    10.104.18.106   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-broker-0                         1/1     Running                  0               133m    10.104.4.247    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-proxy-0                          1/1     Running                  0               133m    10.104.4.246    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-pulsar-init-rc9fs                0/1     Completed                0               133m    10.104.18.105   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-recovery-0                       1/1     Running                  0               133m    10.104.18.112   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-0                      1/1     Running                  0               133m    10.104.4.251    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-1                      1/1     Running                  0               133m    10.104.24.133   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-2                      1/1     Running                  0               132m    10.104.9.130    4am-node14   <none>           <none>

logs:

datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:52:31.468 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/datanode.NewServer.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:82\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:259\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
Show context
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:52:31.598 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/datanode.NewServer.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:82\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:259\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:52:31.601 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datacoord
datacoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datacoord-7d6b659d74-hfk7c
[2023/08/21 03:52:31.605 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:276\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:296\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:105\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:216\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:52:32.325 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:502\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
rootcoord
rootcoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-rootcoord-784b96cbdc-wwvjb
[2023/08/21 03:52:32.514 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:56\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).setClient.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:122\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:182\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:140\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
indexnode
indexnode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-rgw9v
[2023/08/21 03:52:32.548 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:183\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:197\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:194\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:171\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:66\ngithub.com/milvus-io/milvus/cmd/components.(*IndexNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_node.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
indexnode
indexnode
4am-node16
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-kh8tx
[2023/08/21 03:52:36.966 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:183\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:197\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:194\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:171\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:66\ngithub.com/milvus-io/milvus/cmd/components.(*IndexNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_node.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
indexnode
indexnode
4am-node18
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-brkrr
[2023/08/21 03:52:52.348 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:183\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:197\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:194\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:171\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:66\ngithub.com/milvus-io/milvus/cmd/components.(*IndexNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_node.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:53:42.481 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/datanode.NewServer.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:82\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:259\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:53:42.610 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/datanode.NewServer.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:82\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:259\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:53:42.613 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:53:43.340 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:502\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
rootcoord
rootcoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-rootcoord-784b96cbdc-wwvjb
[2023/08/21 03:53:43.528 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:56\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).setClient.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:122\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:182\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:140\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
indexnode
indexnode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-rgw9v
[2023/08/21 03:53:43.565 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:183\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:197\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:194\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:171\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:66\ngithub.com/milvus-io/milvus/cmd/components.(*IndexNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_node.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:53:45.485 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=0] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:53:45.615 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=0] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:53:46.344 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=0] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:53:48.675 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=0] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:54:06.543 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=10] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:54:06.641 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=10] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:54:07.368 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=10] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:54:09.708 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=10] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
rootcoord
rootcoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-rootcoord-784b96cbdc-wwvjb
[2023/08/21 03:54:27.383 +00:00] [ERROR] [msgstream/mq_msgstream.go:122] ["retry func failed"] ["retry time"=0] [error="connection error"] [stack="github.com/milvus-io/milvus/pkg/mq/msgstream.(*mqMsgStream).AsProducer\n\t/go/src/github.com/milvus-io/milvus/pkg/mq/msgstream/mq_msgstream.go:122\ngithub.com/milvus-io/milvus/internal/rootcoord.newDmlChannels\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/dml_channels.go:202\ngithub.com/milvus-io/milvus/internal/rootcoord.newTimeTickSync\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/timeticksync.go:122\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).initInternal\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:442\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Init.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:512\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Init\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:511\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:214\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:140\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:54:36.569 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=20] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:54:36.668 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=20] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:54:37.391 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=20] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:54:39.744 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=20] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:55:06.595 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=30] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:55:06.695 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=30] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:55:07.419 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=30] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:55:09.790 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=30] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:55:36.620 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=40] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:55:36.720 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=40] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:55:37.446 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=40] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:55:39.826 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=40] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:56:06.644 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=50] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:56:06.748 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=50] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:56:07.470 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=50] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:56:09.867 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=50] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node28
lb-helm-ivfsq8-scene-high-milvus-querynode-5cbcc747d9-q9z9z
LOG] [ERROR] 2023-08-21 04:47:56.882 AWSClient [140097193305856] HTTP response code: -1
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 04:48:05.318 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199646] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 04:48:05.318 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199646] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 04:48:32.793 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199685] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 04:48:34.179 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199685] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 04:48:34.179 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199685] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:03:42.917 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321329211052] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:03:45.917 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321329210949] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:09:03.792 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321330018295] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:09:03.792 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321330218570] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:10:36.294 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321330824786] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:11:08.421 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321331226508] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:11:16.421 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321331427421] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:11:30.295 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321331427938] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:11:58.295 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321331629645] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:12:44.294 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199685] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:12:45.797 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321331629696] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:16:10.797 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321332667111] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:16:19.300 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321332871220] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:16:20.296 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321332871216] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:16:35.421 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321332880301] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:16:51.296 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321333081845] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:17:41.298 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321332880699] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:18:11.295 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321333081831] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:19:23.296 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321333954709] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:19:33.922 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321334456221] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:20:57.799 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321334877397] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:20:59.184 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321334878981] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:29:33.933 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321336143094] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:30:00.999 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321336163507] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:30:01.022 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321336163476] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:42:37.423 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321338874018] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:43:48.683 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321339277007] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:44:25.800 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321339277113] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:49:08.304 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321340080814] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:55:03.199 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321340684293] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 06:01:41.311 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321342187087] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 06:03:12.312 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321342430787] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 06:03:41.934 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321342885621] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]

clients log:

Recurrent

image tag: master-20230818-74fb244b argo task: fouramf-server-client-concurrent-5bk4s

server:

NAME                                                              READY   STATUS                   RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lb-helm-ivfsq8-scene-high-etcd-0                                  1/1     Running                  0               133m    10.104.4.250    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-1                                  1/1     Running                  0               133m    10.104.6.30     4am-node13   <none>           <none>
lb-helm-ivfsq8-scene-high-etcd-2                                  1/1     Running                  0               133m    10.104.24.129   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datacoord-7d6b659d74-hfk7c       1/1     Running                  0               133m    10.104.18.108   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg         1/1     Running                  1 (129m ago)    133m    10.104.18.109   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp         1/1     Running                  0               133m    10.104.24.126   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexcoord-5cbbb47d88-rrtdp      1/1     Running                  0               133m    10.104.18.111   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-brkrr        1/1     Running                  0               133m    10.104.14.253   4am-node18   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-kh8tx        1/1     Running                  0               133m    10.104.13.239   4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-rgw9v        1/1     Running                  0               133m    10.104.18.113   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9           1/1     Running                  1 (129m ago)    133m    10.104.24.127   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd      1/1     Running                  1 (129m ago)    133m    10.104.18.107   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-5cbcc747d9-bw8lp       1/1     Running                  0               133m    10.104.24.173   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-5cbcc747d9-gw4xk       1/1     Running                  0               133m    10.104.13.244   4am-node16   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-querynode-5cbcc747d9-q9z9z       1/1     Running                  0               133m    10.104.19.48    4am-node28   <none>           <none>
lb-helm-ivfsq8-scene-high-milvus-rootcoord-784b96cbdc-wwvjb       1/1     Running                  0               133m    10.104.18.110   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-0                                 1/1     Running                  0               133m    10.104.18.116   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-1                                 1/1     Running                  0               133m    10.104.4.2      4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-2                                 1/1     Running                  0               133m    10.104.9.128    4am-node14   <none>           <none>
lb-helm-ivfsq8-scene-high-minio-3                                 1/1     Running                  0               133m    10.104.24.131   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-0                         1/1     Running                  0               133m    10.104.18.118   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-1                         1/1     Running                  0               133m    10.104.4.6      4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-2                         1/1     Running                  0               133m    10.104.1.230    4am-node10   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-bookie-init-trr5k                0/1     Completed                0               133m    10.104.18.106   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-broker-0                         1/1     Running                  0               133m    10.104.4.247    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-proxy-0                          1/1     Running                  0               133m    10.104.4.246    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-pulsar-init-rc9fs                0/1     Completed                0               133m    10.104.18.105   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-recovery-0                       1/1     Running                  0               133m    10.104.18.112   4am-node25   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-0                      1/1     Running                  0               133m    10.104.4.251    4am-node11   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-1                      1/1     Running                  0               133m    10.104.24.133   4am-node29   <none>           <none>
lb-helm-ivfsq8-scene-high-pulsar-zookeeper-2                      1/1     Running                  0               132m    10.104.9.130    4am-node14   <none>           <none>

logs:

datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:52:31.468 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/datanode.NewServer.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:82\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:259\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
Show context
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:52:31.598 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/datanode.NewServer.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:82\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:259\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:52:31.601 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datacoord
datacoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datacoord-7d6b659d74-hfk7c
[2023/08/21 03:52:31.605 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:276\ngithub.com/milvus-io/milvus/internal/datacoord.(*Server).Init\n\t/go/src/github.com/milvus-io/milvus/internal/datacoord/server.go:296\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:105\ngithub.com/milvus-io/milvus/internal/distributed/datacoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/service.go:216\ngithub.com/milvus-io/milvus/cmd/components.(*DataCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_coord.go:49\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:52:32.325 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:502\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
rootcoord
rootcoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-rootcoord-784b96cbdc-wwvjb
[2023/08/21 03:52:32.514 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:56\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).setClient.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:122\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:182\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:140\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
indexnode
indexnode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-rgw9v
[2023/08/21 03:52:32.548 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:183\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:197\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:194\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:171\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:66\ngithub.com/milvus-io/milvus/cmd/components.(*IndexNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_node.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
indexnode
indexnode
4am-node16
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-kh8tx
[2023/08/21 03:52:36.966 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:183\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:197\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:194\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:171\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:66\ngithub.com/milvus-io/milvus/cmd/components.(*IndexNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_node.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
indexnode
indexnode
4am-node18
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-brkrr
[2023/08/21 03:52:52.348 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=0] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:183\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:197\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:194\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:171\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:66\ngithub.com/milvus-io/milvus/cmd/components.(*IndexNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_node.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:53:42.481 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/datanode.NewServer.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:82\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:259\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:53:42.610 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/datanode.NewServer.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:82\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:259\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:53:42.613 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:135\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:53:43.340 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:57\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:502\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
rootcoord
rootcoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-rootcoord-784b96cbdc-wwvjb
[2023/08/21 03:53:43.528 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/distributed/datacoord/client.NewClient\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:56\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).setClient.func1\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:122\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:182\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:140\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
indexnode
indexnode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-indexnode-d4b6f75cb-rgw9v
[2023/08/21 03:53:43.565 +00:00] [ERROR] [sessionutil/session_util.go:234] ["retry func failed"] ["retry time"=10] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/util/sessionutil.NewSession\n\t/go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:234\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).initSession\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:183\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init.func1\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:197\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/indexnode.(*IndexNode).Init\n\t/go/src/github.com/milvus-io/milvus/internal/indexnode/indexnode.go:194\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:171\ngithub.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:66\ngithub.com/milvus-io/milvus/cmd/components.(*IndexNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/index_node.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:53:45.485 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=0] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:53:45.615 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=0] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:53:46.344 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=0] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:53:48.675 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=0] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:54:06.543 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=10] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:54:06.641 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=10] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:54:07.368 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=10] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:54:09.708 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=10] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
rootcoord
rootcoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-rootcoord-784b96cbdc-wwvjb
[2023/08/21 03:54:27.383 +00:00] [ERROR] [msgstream/mq_msgstream.go:122] ["retry func failed"] ["retry time"=0] [error="connection error"] [stack="github.com/milvus-io/milvus/pkg/mq/msgstream.(*mqMsgStream).AsProducer\n\t/go/src/github.com/milvus-io/milvus/pkg/mq/msgstream/mq_msgstream.go:122\ngithub.com/milvus-io/milvus/internal/rootcoord.newDmlChannels\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/dml_channels.go:202\ngithub.com/milvus-io/milvus/internal/rootcoord.newTimeTickSync\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/timeticksync.go:122\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).initInternal\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:442\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Init.func2\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:512\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:68\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:59\ngithub.com/milvus-io/milvus/internal/rootcoord.(*Core).Init\n\t/go/src/github.com/milvus-io/milvus/internal/rootcoord/root_coord.go:511\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:214\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/service.go:140\ngithub.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:54:36.569 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=20] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:54:36.668 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=20] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:54:37.391 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=20] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:54:39.744 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=20] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:55:06.595 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=30] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:55:06.695 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=30] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:55:07.419 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=30] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:55:09.790 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=30] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:55:36.620 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=40] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:55:36.720 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=40] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:55:37.446 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=40] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:55:39.826 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=40] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node29
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-hdmdp
[2023/08/21 03:56:06.644 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=50] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
datanode
datanode
4am-node25
lb-helm-ivfsq8-scene-high-milvus-datanode-9755c8bdd-2chsg
[2023/08/21 03:56:06.748 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=50] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176 github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:272\ngithub.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/datanode/service.go:176\ngithub.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/data_node.go:52\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
proxy
proxy
4am-node29
lb-helm-ivfsq8-scene-high-milvus-proxy-79d944cff6-ctcm9
[2023/08/21 03:56:07.470 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=50] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:518\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:342\ngithub.com/milvus-io/milvus/cmd/components.(*Proxy).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/proxy.go:51\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 03:56:09.867 +00:00] [ERROR] [componentutil/componentutil.go:58] ["retry func failed"] ["retry time"=50] [error="err: find no available rootcoord, check rootcoord state\n, /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:359 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:132 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:146 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:34 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates.func1\n/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:40 github.com/milvus-io/milvus/pkg/util/retry.Do\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73 github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92 github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n"] [stack="github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:58\ngithub.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy\n\t/go/src/github.com/milvus-io/milvus/internal/util/componentutil/componentutil.go:73\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:153\ngithub.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/querycoord/service.go:92\ngithub.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/query_coord.go:53\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:112"]
querynode
querynode
4am-node28
lb-helm-ivfsq8-scene-high-milvus-querynode-5cbcc747d9-q9z9z
LOG] [ERROR] 2023-08-21 04:47:56.882 AWSClient [140097193305856] HTTP response code: -1
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 04:48:05.318 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199646] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 04:48:05.318 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199646] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 04:48:32.793 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199685] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 04:48:34.179 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199685] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 04:48:34.179 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199685] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:03:42.917 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321329211052] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:03:45.917 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321329210949] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:09:03.792 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321330018295] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:09:03.792 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321330218570] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:10:36.294 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321330824786] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:11:08.421 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321331226508] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:11:16.421 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321331427421] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:11:30.295 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321331427938] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:11:58.295 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321331629645] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:12:44.294 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321211199685] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:12:45.797 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321331629696] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:16:10.797 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321332667111] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:16:19.300 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321332871220] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:16:20.296 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321332871216] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:16:35.421 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321332880301] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:16:51.296 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321333081845] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:17:41.298 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321332880699] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:18:11.295 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321333081831] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:19:23.296 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321333954709] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:19:33.922 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321334456221] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:20:57.799 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321334877397] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:20:59.184 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321334878981] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:29:33.933 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321336143094] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:30:00.999 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321336163507] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:30:01.022 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321336163476] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:42:37.423 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321338874018] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:43:48.683 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321339277007] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:44:25.800 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321339277113] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:49:08.304 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321340080814] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 05:55:03.199 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321340684293] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 06:01:41.311 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321342187087] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 06:03:12.312 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321342430787] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]
querycoord
querycoord
4am-node25
lb-helm-ivfsq8-scene-high-milvus-querycoord-585b75bcc9-5lkpd
[2023/08/21 06:03:41.934 +00:00] [ERROR] [meta/coordinator_broker.go:251] ["failed to fetch index meta"] [collection=443702321342885621] [error="context canceled"] [stack="github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeIndex\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:251\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).loadSegment\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:289\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).executeSegmentAction\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:214\ngithub.com/milvus-io/milvus/internal/querycoordv2/task.(*Executor).Execute.func1\n\t/go/src/github.com/milvus-io/milvus/internal/querycoordv2/task/executor.go:113"]

clients log: 截屏2023-08-21 14 07 32

problem: lots of load collection timeout.

root cause: here comes hunderd of loading collection operation in parallel, which makes collection_observer stuck at observePartitionLoadStatus for serval minutes. for some collection, it loaded all channel and segments, but blocked by collection_observer, couldn't become loaded before timeout.

fix way: to reduce the cost overhead in observePartitionLoadStatus

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

milvus-io / milvus