search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.51k stars 2.83k forks source link

[Bug]: [benchmark][cluster] QueryNode restart and search raise error: fail to search on all shard leaders, err=fail to Search, QueryNode ID=3, reason=query node 12 is not ready #19543

Closed wangting0128 closed 1 year ago

wangting0128 commented 1 year ago

Is there an existing issue for this?

Environment

- Milvus version:master-20220928-a8cabbfa
- Deployment mode(standalone or cluster):cluster
- SDK version(e.g. pymilvus v2.0.0rc2):2.2.0.dev32
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo link:fouramf-cron-1664380800 test case: test_search_time

client log: test_search_time_2.zip

[2022-09-28 17:25:09,849 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_OQRgrfgf): 49825054 (base.py:237)
[2022-09-28 17:25:10,971 -  INFO - fouram]: [Base] Start inserting, ids: 49900000 - 49949999, data size: 50,000,000 (base.py:114)
[2022-09-28 17:25:11,975 -  INFO - fouram]: [Time] Collection.insert run in 1.0037s (api_request.py:29)
[2022-09-28 17:25:11,977 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_OQRgrfgf): 49874987 (base.py:237)
[2022-09-28 17:25:12,192 -  INFO - fouram]: [Base] Start inserting, ids: 49950000 - 49999999, data size: 50,000,000 (base.py:114)
[2022-09-28 17:25:13,233 -  INFO - fouram]: [Time] Collection.insert run in 1.0399s (api_request.py:29)
[2022-09-28 17:25:13,235 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_OQRgrfgf): 49924982 (base.py:237)
[2022-09-28 17:25:13,300 -  INFO - fouram]: [Base] Total time of insert: 1150.6605s, average number of vector bars inserted per second: 43453.3036, average time to insert 50000 vectors per time: 1.1507s (base.py:172)
[2022-09-28 17:25:13,301 -  INFO - fouram]: [Base] Start flush collection fouram_OQRgrfgf (base.py:91)
[2022-09-28 17:25:15,816 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_OQRgrfgf): 50000000 (base.py:237)
[2022-09-28 17:25:15,820 -  INFO - fouram]: [Base] Params of index: {'index_type': 'IVF_FLAT', 'metric_type': 'L2', 'params': {'nlist': 2048}} (base.py:221)
[2022-09-28 17:25:15,821 -  INFO - fouram]: [Base] Start build index of IVF_FLAT, params:{'index_type': 'IVF_FLAT', 'metric_type': 'L2', 'params': {'nlist': 2048}} (base.py:215)
[2022-09-28 17:25:15,824 -  INFO - fouram]: [Time] Index run in 0.0029s (api_request.py:29)
[2022-09-28 17:25:15,824 -  INFO - fouram]: [CommonCases] RT of build index IVF_FLAT: 0.0029s (common_cases.py:80)
[2022-09-28 17:25:15,827 -  INFO - fouram]: [Base] Params of index: {'index_type': 'IVF_FLAT', 'metric_type': 'L2', 'params': {'nlist': 2048}} (base.py:221)
[2022-09-28 17:25:15,827 -  INFO - fouram]: [CommonCases] Prepare index IVF_FLAT done. (common_cases.py:83)
[2022-09-28 17:25:15,827 -  INFO - fouram]: [Base] Start load collection fouram_OQRgrfgf, replica_number:1 (base.py:95)
[2022-09-28 17:28:31,366 -  INFO - fouram]: [Time] Collection.load run in 195.5378s (api_request.py:29)
[2022-09-28 17:28:32,889 -  INFO - fouram]: [PerfTemplate] Actual parameters used: {'collection_params': {'other_fields': []}, 'load_params': {}, 'search_params': {'nq': 1, 'param': {'metric_type': 'L2', 'params': {'nprobe': 8}}, 'top_k': 1, 'expr': None}, 'dataset_params': {'dataset_name': 'sift', 'dim': 128, 'dataset_size': 50000000, 'ni_per': 50000, 'metric_type': 'L2', 'req_run_counts': 30}, 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 2048}}} (performance_template.py:57)
[2022-09-28 17:28:32,889 -  INFO - fouram]: [Base] Params of search: nq:1, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'nprobe': 8}}, limit:1, expr:"None" (base.py:261)
[2022-09-28 17:28:34,159 - ERROR - fouram]: Traceback (most recent call last):
  File "/src/fouram/client/util/api_request.py", line 21, in inner_wrapper
    res = func(*args, **kwargs)
  File "/src/fouram/client/util/api_request.py", line 57, in api_request
    return func(*arg, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 717, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 113, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 139, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 89, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 51, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 472, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 436, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 427, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=3, reason=query node 12 is not ready)>
 (api_request.py:35)
[2022-09-28 17:28:34,160 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=3, reason=query node 12 is not ready)> (api_request.py:36)
[2022-09-28 17:28:34,160 - ERROR - fouram]: [CheckFunc] Request check failed, response:<MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=3, reason=query node 12 is not ready)> (func_check.py:40)
[2022-09-28 17:28:34,160 - ERROR - fouram]: [Search] Search raise error:  (common_cases.py:401)
[2022-09-28 17:28:34,161 -  INFO - fouram]: [PerfTemplate] Actual parameters used: {'collection_params': {'other_fields': []}, 'load_params': {}, 'search_params': {'nq': 1, 'param': {'metric_type': 'L2', 'params': {'nprobe': 32}}, 'top_k': 1, 'expr': None}, 'dataset_params': {'dataset_name': 'sift', 'dim': 128, 'dataset_size': 50000000, 'ni_per': 50000, 'metric_type': 'L2', 'req_run_counts': 30}, 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 2048}}} (performance_template.py:57)
[2022-09-28 17:28:34,161 -  INFO - fouram]: [Base] Params of search: nq:1, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'nprobe': 32}}, limit:1, expr:"None" (base.py:261)
[2022-09-28 17:28:34,164 - ERROR - fouram]: Traceback (most recent call last):
  File "/src/fouram/client/util/api_request.py", line 21, in inner_wrapper
    res = func(*args, **kwargs)
  File "/src/fouram/client/util/api_request.py", line 57, in api_request
    return func(*arg, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 717, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 113, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 139, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 89, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 51, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 472, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 436, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 427, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Invalid shard leader)>
 (api_request.py:35)
[2022-09-28 17:28:34,165 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=Invalid shard leader)> (api_request.py:36)
[2022-09-28 17:28:34,165 - ERROR - fouram]: [CheckFunc] Request check failed, response:<MilvusException: (code=1, message=Invalid shard leader)> (func_check.py:40)
[2022-09-28 17:28:34,165 - ERROR - fouram]: [Search] Search raise error:  (common_cases.py:401)

server:

[2022-09-28 17:29:22,166 -  INFO - fouram]: [Base] Deploy initial state: 
I0928 16:54:58.791830    3768 request.go:665] Waited for 1.168814772s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/admissionregistration.k8s.io/v1?timeout=32s
NAME                                                              READY   STATUS        RESTARTS        AGE     IP             NODE         NOMINATED NODE   READINESS GATES
fouramf-cron-1664380800-86-5517-etcd-0                            1/1     Running       0               5m54s   10.104.1.59    4am-node10   <none>           <none>
fouramf-cron-1664380800-86-5517-etcd-1                            1/1     Running       0               5m53s   10.104.9.23    4am-node14   <none>           <none>
fouramf-cron-1664380800-86-5517-etcd-2                            1/1     Running       0               5m53s   10.104.4.85    4am-node11   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-datacoord-8548548dftp4lx   1/1     Running       1 (112s ago)    5m54s   10.104.6.161   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-datanode-b6986d7f-c2bgf    1/1     Running       1 (2m21s ago)   5m53s   10.104.6.165   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-indexcoord-8454896ftw762   1/1     Running       1 (112s ago)    5m54s   10.104.6.160   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-indexnode-d7c66c4b-mvnqx   1/1     Running       0               5m53s   10.104.9.14    4am-node14   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-proxy-dbd9d95d9-72j55      1/1     Running       1 (112s ago)    5m54s   10.104.6.157   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-querycoord-844d7455q68p9   1/1     Running       1 (2m21s ago)   5m54s   10.104.6.164   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-querynode-5ff9694f74655n   1/1     Running       0               5m53s   10.104.5.232   4am-node12   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-rootcoord-5668f8cb4frfh5   1/1     Running       1 (113s ago)    5m54s   10.104.6.155   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-minio-0                           1/1     Running       0               5m53s   10.104.1.63    4am-node10   <none>           <none>
fouramf-cron-1664380800-86-5517-minio-1                           1/1     Running       0               5m53s   10.104.4.86    4am-node11   <none>           <none>
fouramf-cron-1664380800-86-5517-minio-2                           1/1     Running       0               5m53s   10.104.6.178   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-minio-3                           1/1     Running       0               5m53s   10.104.5.241   4am-node12   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-bookie-0                   1/1     Running       0               5m54s   10.104.5.239   4am-node12   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-bookie-1                   1/1     Running       0               5m53s   10.104.1.62    4am-node10   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-bookie-2                   1/1     Running       0               5m53s   10.104.4.87    4am-node11   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-bookie-init-v4qwk          0/1     Completed     0               5m54s   10.104.9.11    4am-node14   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-broker-0                   1/1     Running       0               5m53s   10.104.1.47    4am-node10   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-proxy-0                    1/1     Running       0               5m54s   10.104.9.12    4am-node14   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-pulsar-init-cmg4s          0/1     Completed     0               5m54s   10.104.9.10    4am-node14   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-recovery-0                 1/1     Running       0               5m54s   10.104.6.156   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-zookeeper-0                1/1     Running       0               5m54s   10.104.5.237   4am-node12   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-zookeeper-1                1/1     Running       0               4m49s   10.104.1.67    4am-node10   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-zookeeper-2                1/1     Running       0               4m12s   10.104.9.25    4am-node14   <none>           <none> (base.py:120)
[2022-09-28 17:29:22,166 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'STATUS|fouramf-cron-1664380800-86-5517'  (util_cmd.py:14)
[2022-09-28 17:29:28,125 -  INFO - fouram]: [CliClient] pod details of release(fouramf-cron-1664380800-86-5517): 
 I0928 17:29:23.429941    6302 request.go:665] Waited for 1.15844599s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s
NAME                                                              READY   STATUS      RESTARTS        AGE     IP             NODE         NOMINATED NODE   READINESS GATES
fouramf-cron-1664380800-86-5517-etcd-0                            1/1     Running     0               40m     10.104.1.59    4am-node10   <none>           <none>
fouramf-cron-1664380800-86-5517-etcd-1                            1/1     Running     0               40m     10.104.9.23    4am-node14   <none>           <none>
fouramf-cron-1664380800-86-5517-etcd-2                            1/1     Running     0               40m     10.104.4.85    4am-node11   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-datacoord-8548548dftp4lx   1/1     Running     1 (36m ago)     40m     10.104.6.161   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-datanode-b6986d7f-c2bgf    1/1     Running     1 (36m ago)     40m     10.104.6.165   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-indexcoord-8454896ftw762   1/1     Running     1 (36m ago)     40m     10.104.6.160   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-indexnode-d7c66c4b-mvnqx   1/1     Running     0               40m     10.104.9.14    4am-node14   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-proxy-dbd9d95d9-72j55      1/1     Running     1 (36m ago)     40m     10.104.6.157   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-querycoord-844d7455q68p9   1/1     Running     1 (36m ago)     40m     10.104.6.164   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-querynode-5ff9694f74655n   0/1     Running     1 (55s ago)     40m     10.104.5.232   4am-node12   <none>           <none>
fouramf-cron-1664380800-86-5517-milvus-rootcoord-5668f8cb4frfh5   1/1     Running     1 (36m ago)     40m     10.104.6.155   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-minio-0                           1/1     Running     0               40m     10.104.1.63    4am-node10   <none>           <none>
fouramf-cron-1664380800-86-5517-minio-1                           1/1     Running     0               40m     10.104.4.86    4am-node11   <none>           <none>
fouramf-cron-1664380800-86-5517-minio-2                           1/1     Running     0               40m     10.104.6.178   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-minio-3                           1/1     Running     0               40m     10.104.5.241   4am-node12   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-bookie-0                   1/1     Running     0               40m     10.104.5.239   4am-node12   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-bookie-1                   1/1     Running     0               40m     10.104.1.62    4am-node10   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-bookie-2                   1/1     Running     0               40m     10.104.4.87    4am-node11   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-bookie-init-v4qwk          0/1     Completed   0               40m     10.104.9.11    4am-node14   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-broker-0                   1/1     Running     0               40m     10.104.1.47    4am-node10   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-proxy-0                    1/1     Running     0               40m     10.104.9.12    4am-node14   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-pulsar-init-cmg4s          0/1     Completed   0               40m     10.104.9.10    4am-node14   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-recovery-0                 1/1     Running     0               40m     10.104.6.156   4am-node13   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-zookeeper-0                1/1     Running     0               40m     10.104.5.237   4am-node12   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-zookeeper-1                1/1     Running     0               39m     10.104.1.67    4am-node10   <none>           <none>
fouramf-cron-1664380800-86-5517-pulsar-zookeeper-2                1/1     Running     0               38m     10.104.9.25    4am-node14   <none>           <none> (cli_client.py:123)
[2022-09-28 17:29:28,127 -  INFO - fouram]: [Base] Start deleting services: fouramf-cron-1664380800-86-5517 (base.py:123)
[2022-09-28 17:29:28,127 -  INFO - fouram]: [Cmd Exe]  kubectl get pvc  -n qa-milvus  | grep -E 'STATUS|fouramf-cron-1664380800-86-5517'  (util_cmd.py:14)
[2022-09-28 17:29:34,031 -  INFO - fouram]: [CliClient] pvc storage class of release(fouramf-cron-1664380800-86-5517): 
 I0928 17:29:29.395260    6355 request.go:665] Waited for 1.1414948s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/node.k8s.io/v1?timeout=32s
NAME                                                                                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
data-fouramf-cron-1664380800-86-5517-etcd-0                                                                Bound    pvc-86480ff9-a146-49d1-a580-f6ca926bdf56   10Gi       RWO            local-path        40m
data-fouramf-cron-1664380800-86-5517-etcd-1                                                                Bound    pvc-9d994fcf-20f6-48d2-b3b7-7d43ca81bca7   10Gi       RWO            local-path        40m
data-fouramf-cron-1664380800-86-5517-etcd-2                                                                Bound    pvc-0bf25eba-5e67-424d-a6ee-efe6c438a073   10Gi       RWO            local-path        40m
export-fouramf-cron-1664380800-86-5517-minio-0                                                             Bound    pvc-2e5902b5-1b66-4c5d-8454-51cb0b64e348   500Gi      RWO            local-path        40m
export-fouramf-cron-1664380800-86-5517-minio-1                                                             Bound    pvc-2f26ccdb-059c-41a9-8824-e05373af1562   500Gi      RWO            local-path        40m
export-fouramf-cron-1664380800-86-5517-minio-2                                                             Bound    pvc-a113ea17-c946-4bd9-aa52-d34358150d46   500Gi      RWO            local-path        40m
export-fouramf-cron-1664380800-86-5517-minio-3                                                             Bound    pvc-0c69f61c-1272-4075-a8b9-a41ee13fd7dc   500Gi      RWO            local-path        40m
fouramf-cron-1664380800-86-5517-pulsar-bookie-journal-fouramf-cron-1664380800-86-5517-pulsar-bookie-0      Bound    pvc-07231de8-42cb-4e5a-a559-8e93799b843e   100Gi      RWO            local-path        40m
fouramf-cron-1664380800-86-5517-pulsar-bookie-journal-fouramf-cron-1664380800-86-5517-pulsar-bookie-1      Bound    pvc-c1672c8c-3319-40fa-a7af-beeb53912371   100Gi      RWO            local-path        40m
fouramf-cron-1664380800-86-5517-pulsar-bookie-journal-fouramf-cron-1664380800-86-5517-pulsar-bookie-2      Bound    pvc-b84c02a1-c361-4f74-b805-7b27497018fb   100Gi      RWO            local-path        40m
fouramf-cron-1664380800-86-5517-pulsar-bookie-ledgers-fouramf-cron-1664380800-86-5517-pulsar-bookie-0      Bound    pvc-43a46a79-244d-4145-bf4a-8885175c884b   200Gi      RWO            local-path        40m
fouramf-cron-1664380800-86-5517-pulsar-bookie-ledgers-fouramf-cron-1664380800-86-5517-pulsar-bookie-1      Bound    pvc-205d93d4-9f8e-477f-8e4e-24b0466c68f1   200Gi      RWO            local-path        40m
fouramf-cron-1664380800-86-5517-pulsar-bookie-ledgers-fouramf-cron-1664380800-86-5517-pulsar-bookie-2      Bound    pvc-bd120932-ba0d-482b-ade1-b8a93b7fda0d   200Gi      RWO            local-path        40m
fouramf-cron-1664380800-86-5517-pulsar-zookeeper-data-fouramf-cron-1664380800-86-5517-pulsar-zookeeper-0   Bound    pvc-4f4ef09a-97b7-4bce-80fe-9e4f058b5b13   20Gi       RWO            local-path        40m
fouramf-cron-1664380800-86-5517-pulsar-zookeeper-data-fouramf-cron-1664380800-86-5517-pulsar-zookeeper-1   Bound    pvc-4c6015c3-6027-4581-8dca-17ee106be1b7   20Gi       RWO            local-path        39m
fouramf-cron-1664380800-86-5517-pulsar-zookeeper-data-fouramf-cron-1664380800-86-5517-pulsar-zookeeper-2   Bound    pvc-cc3e252a-1012-4e6c-b88c-14d170c380d4   20Gi       RWO            local-path        38m (cli_client.py:131)

Expected Behavior

No response

Steps To Reproduce

1、create collection
2、build index of ivf_flat
3、insert 50m vectors
4、flush collection
5、build index with the same params
6、load collection
7、search 《- raise error

Milvus Log

No response

Anything else?

client params:

{
     "collection_params": {
          "other_fields": []
     },
     "load_params": {},
     "search_params": {
          "top_k": [
               1,
               10,
               100,
               1000
          ],
          "nq": [
               1,
               10,
               100,
               200,
               500,
               1000,
               1200
          ],
          "search_param": {
               "nprobe": [
                    8,
                    32
               ]
          }
     },
     "dataset_params": {
          "dataset_name": "sift",
          "dim": 128,
          "dataset_size": 50000000,
          "ni_per": 50000,
          "metric_type": "L2",
          "req_run_counts": 30
     },
     "index_params": {
          "index_type": "IVF_FLAT",
          "index_param": {
               "nlist": 2048
          }
     }
}

server deploy params: queryNode.resources.limits.cpu=16.0,queryNode.resources.limits.memory=64Gi,queryNode.resources.requests.cpu=9.0,queryNode.resources.requests.memory=33Gi,indexNode.resources.limits.cpu=16.0,indexNode.resources.limits.memory=64Gi,indexNode.resources.requests.cpu=9.0,indexNode.resources.requests.memory=33Gi,dataNode.resources.limits.cpu=16.0,dataNode.resources.limits.memory=64Gi,dataNode.resources.requests.cpu=9.0,dataNode.resources.requests.memory=33Gi,cluster.enabled=True,pulsar.bookkeeper.volumes.journal.storageClassName=local-path,pulsar.bookkeeper.volumes.ledgers.storageClassName=local-path,pulsar.zookeeper.volumes.data.storageClassName=local-path,kafka.persistence.storageClass=local-path,minio.persistence.storageClass=local-path,minio.metrics.serviceMonitor.enabled=False,etcd.global.storageClass=local-path,etcd.metrics.enabled=True,etcd.metrics.podMonitor.enabled=True,metrics.serviceMonitor.enabled=True,image.all.repository=harbor.milvus.io/dockerhub/milvusdb/milvus,image.all.tag=master-20220928-a8cabbfa

yanliang567 commented 1 year ago

/assign @congqixia /unassign

wangting0128 commented 1 year ago

argo link:fouramf-cron-1664467200

test case: test_search_time log: test_search_time_2.zip

congqixia commented 1 year ago

Hi @wangting0128 ,does this problem still persist in recent runs ?

wangting0128 commented 1 year ago

argo task: fouramf-cron-1665504000 test case: test_search_time

image tag:master-20221011-0a77561e client pod:fouramf-cron-1665504000-2268600349

standalone server:

[2022-10-11 16:53:01,137 -  INFO - fouram]: [Base] Deploy initial state: 
I1011 16:10:06.349376     357 request.go:665] Waited for 1.175620138s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/networking.k8s.io/v1?timeout=32s
NAME                                                              READY   STATUS             RESTARTS         AGE     IP             NODE         NOMINATED NODE   READINESS GATES
fouramf-cron-1665504000-66-9908-etcd-0                            1/1     Running            0                9m32s   10.104.6.146   4am-node13   <none>           <none>
fouramf-cron-1665504000-66-9908-milvus-standalone-8585c877nmw59   1/1     Running            0                9m32s   10.104.5.199   4am-node12   <none>           <none>
fouramf-cron-1665504000-66-9908-minio-54f7c45b5d-z9l6n            1/1     Running            0                9m32s   10.104.5.200   4am-node12   <none>           <none> (base.py:120)
[2022-10-11 16:53:01,138 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'STATUS|fouramf-cron-1665504000-66-9908'  (util_cmd.py:14)
[2022-10-11 16:53:07,164 -  INFO - fouram]: [CliClient] pod details of release(fouramf-cron-1665504000-66-9908): 
 I1011 16:53:02.421014    3077 request.go:665] Waited for 1.155193157s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/chaos-mesh.org/v1alpha1?timeout=32s
NAME                                                              READY   STATUS             RESTARTS         AGE     IP             NODE         NOMINATED NODE   READINESS GATES
fouramf-cron-1665504000-66-9908-etcd-0                            1/1     Running            0                52m     10.104.6.146   4am-node13   <none>           <none>
fouramf-cron-1665504000-66-9908-milvus-standalone-8585c877nmw59   1/1     Running            1 (2m46s ago)    52m     10.104.5.199   4am-node12   <none>           <none>
fouramf-cron-1665504000-66-9908-minio-54f7c45b5d-z9l6n            1/1     Running            0                52m     10.104.5.200   4am-node12   <none>           <none> (cli_client.py:123)

client log:

[2022-10-11 16:44:47,462 -  INFO - fouram]: [CommonCases] RT of build index IVF_FLAT: 0.0019s (common_cases.py:80)
[2022-10-11 16:44:47,464 -  INFO - fouram]: [Base] Params of index: {'index_type': 'IVF_FLAT', 'metric_type': 'L2', 'params': {'nlist': 2048}} (base.py:221)
[2022-10-11 16:44:47,464 -  INFO - fouram]: [CommonCases] Prepare index IVF_FLAT done. (common_cases.py:83)
[2022-10-11 16:44:47,464 -  INFO - fouram]: [Base] Start load collection fouram_LJYyBpZb, replica_number:1 (base.py:95)
[2022-10-11 16:50:17,815 -  INFO - fouram]: [Time] Collection.load run in 330.3499s (api_request.py:29)
[2022-10-11 16:50:19,883 -  INFO - fouram]: [PerfTemplate] Actual parameters used: {'collection_params': {'other_fields': []}, 'load_params': {}, 'search_params': {'nq': 1, 'param': {'metric_type': 'L2', 'params': {'nprobe': 8}}, 'top_k': 1, 'expr': None}, 'dataset_params': {'dataset_name': 'sift', 'dim': 128, 'dataset_size': 50000000, 'ni_per': 50000, 'metric_type': 'L2', 'req_run_counts': 30}, 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 2048}}} (performance_template.py:57)
[2022-10-11 16:50:19,884 -  INFO - fouram]: [Base] Params of search: nq:1, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'nprobe': 8}}, limit:1, expr:"None" (base.py:261)
[2022-10-11 16:52:54,408 - ERROR - fouram]: Traceback (most recent call last):
  File "/src/fouram/client/util/api_request.py", line 21, in inner_wrapper
    res = func(*args, **kwargs)
  File "/src/fouram/client/util/api_request.py", line 57, in api_request
    return func(*arg, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 717, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 113, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 139, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 89, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 51, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 472, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 436, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 427, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=collection:fouram_LJYyBpZb or partition:[] not loaded into memory when search)>
 (api_request.py:35)
[2022-10-11 16:52:54,408 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=collection:fouram_LJYyBpZb or partition:[] not loaded into memory when search)> (api_request.py:36)

cluster server:

[2022-10-11 17:28:00,560 -  INFO - fouram]: [Base] Deploy initial state: 
I1011 16:57:39.293634    3416 request.go:665] Waited for 1.169029684s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/batch/v1beta1?timeout=32s
NAME                                                              READY   STATUS             RESTARTS         AGE     IP             NODE         NOMINATED NODE   READINESS GATES
fouramf-cron-1665504000-20-9119-etcd-0                            1/1     Running            0                4m11s   10.104.4.49    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-etcd-1                            1/1     Running            0                4m11s   10.104.9.16    4am-node14   <none>           <none>
fouramf-cron-1665504000-20-9119-etcd-2                            1/1     Running            0                4m11s   10.104.1.24    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-datacoord-647d785dfk44zf   1/1     Running            0                4m11s   10.104.5.224   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-datanode-6d479db79cxp5qc   1/1     Running            0                4m11s   10.104.1.19    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-indexcoord-7fb9fc68lc6xc   1/1     Running            0                4m11s   10.104.1.20    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-indexnode-697f44494rh4ns   1/1     Running            0                4m11s   10.104.5.225   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-proxy-57c886474f-78j7x     1/1     Running            0                4m11s   10.104.5.226   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-querycoord-58c8b8cdkdslm   1/1     Running            0                4m11s   10.104.6.173   4am-node13   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-querynode-b55b5dd6dtzzn4   1/1     Running            0                4m11s   10.104.6.174   4am-node13   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-rootcoord-8c975f49b522c8   1/1     Running            0                4m11s   10.104.1.16    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-minio-0                           1/1     Running            0                4m11s   10.104.4.46    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-minio-1                           1/1     Running            0                4m11s   10.104.5.230   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-minio-2                           1/1     Running            0                4m11s   10.104.1.22    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-minio-3                           1/1     Running            0                4m10s   10.104.6.178   4am-node13   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-bookie-0                   1/1     Running            0                4m11s   10.104.4.48    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-bookie-1                   1/1     Running            0                4m11s   10.104.5.231   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-bookie-2                   1/1     Running            0                4m10s   10.104.6.179   4am-node13   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-bookie-init-9pl68          0/1     Completed          0                4m11s   10.104.4.38    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-broker-0                   1/1     Running            0                4m11s   10.104.4.40    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-proxy-0                    1/1     Running            0                4m11s   10.104.1.17    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-pulsar-init-xpp9j          0/1     Completed          0                4m11s   10.104.1.18    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-recovery-0                 1/1     Running            0                4m11s   10.104.4.39    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-zookeeper-0                1/1     Running            0                4m11s   10.104.4.45    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-zookeeper-1                1/1     Running            0                3m33s   10.104.5.233   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-zookeeper-2                1/1     Running            0                2m58s   10.104.9.18    4am-node14   <none>           <none> (base.py:120)
[2022-10-11 17:28:00,560 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'STATUS|fouramf-cron-1665504000-20-9119'  (util_cmd.py:14)
[2022-10-11 17:28:06,566 -  INFO - fouram]: [CliClient] pod details of release(fouramf-cron-1665504000-20-9119): 
 I1011 17:28:01.827927    5801 request.go:665] Waited for 1.161576177s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/acme.cert-manager.io/v1?timeout=32s
NAME                                                              READY   STATUS             RESTARTS          AGE   IP             NODE         NOMINATED NODE   READINESS GATES
fouramf-cron-1665504000-20-9119-etcd-0                            1/1     Running            0                 34m   10.104.4.49    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-etcd-1                            1/1     Running            0                 34m   10.104.9.16    4am-node14   <none>           <none>
fouramf-cron-1665504000-20-9119-etcd-2                            1/1     Running            0                 34m   10.104.1.24    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-datacoord-647d785dfk44zf   1/1     Running            0                 34m   10.104.5.224   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-datanode-6d479db79cxp5qc   1/1     Running            0                 34m   10.104.1.19    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-indexcoord-7fb9fc68lc6xc   1/1     Running            0                 34m   10.104.1.20    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-indexnode-697f44494rh4ns   1/1     Running            0                 34m   10.104.5.225   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-proxy-57c886474f-78j7x     1/1     Running            0                 34m   10.104.5.226   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-querycoord-58c8b8cdkdslm   1/1     Running            0                 34m   10.104.6.173   4am-node13   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-querynode-b55b5dd6dtzzn4   0/1     Running            1 (60s ago)       34m   10.104.6.174   4am-node13   <none>           <none>
fouramf-cron-1665504000-20-9119-milvus-rootcoord-8c975f49b522c8   1/1     Running            0                 34m   10.104.1.16    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-minio-0                           1/1     Running            0                 34m   10.104.4.46    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-minio-1                           1/1     Running            0                 34m   10.104.5.230   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-minio-2                           1/1     Running            0                 34m   10.104.1.22    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-minio-3                           1/1     Running            0                 34m   10.104.6.178   4am-node13   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-bookie-0                   1/1     Running            0                 34m   10.104.4.48    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-bookie-1                   1/1     Running            0                 34m   10.104.5.231   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-bookie-2                   1/1     Running            0                 34m   10.104.6.179   4am-node13   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-bookie-init-9pl68          0/1     Completed          0                 34m   10.104.4.38    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-broker-0                   1/1     Running            0                 34m   10.104.4.40    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-proxy-0                    1/1     Running            0                 34m   10.104.1.17    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-pulsar-init-xpp9j          0/1     Completed          0                 34m   10.104.1.18    4am-node10   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-recovery-0                 1/1     Running            0                 34m   10.104.4.39    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-zookeeper-0                1/1     Running            0                 34m   10.104.4.45    4am-node11   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-zookeeper-1                1/1     Running            0                 33m   10.104.5.233   4am-node12   <none>           <none>
fouramf-cron-1665504000-20-9119-pulsar-zookeeper-2                1/1     Running            0                 33m   10.104.9.18    4am-node14   <none>           <none> (cli_client.py:123)

client log:

[2022-10-11 17:25:10,364 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_5DS5tURl): 49924982 (base.py:237)
[2022-10-11 17:25:10,421 -  INFO - fouram]: [Base] Total time of insert: 1029.7726s, average number of vector bars inserted per second: 48554.409, average time to insert 50000 vectors per time: 1.0298s (base.py:172)
[2022-10-11 17:25:10,422 -  INFO - fouram]: [Base] Start flush collection fouram_5DS5tURl (base.py:91)
[2022-10-11 17:25:12,962 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_5DS5tURl): 50000000 (base.py:237)
[2022-10-11 17:25:12,965 -  INFO - fouram]: [Base] Params of index: {'index_type': 'IVF_FLAT', 'metric_type': 'L2', 'params': {'nlist': 2048}} (base.py:221)
[2022-10-11 17:25:12,965 -  INFO - fouram]: [Base] Start build index of IVF_FLAT, params:{'index_type': 'IVF_FLAT', 'metric_type': 'L2', 'params': {'nlist': 2048}} (base.py:215)
[2022-10-11 17:25:12,968 -  INFO - fouram]: [Time] Index run in 0.0029s (api_request.py:29)
[2022-10-11 17:25:12,968 -  INFO - fouram]: [CommonCases] RT of build index IVF_FLAT: 0.0029s (common_cases.py:80)
[2022-10-11 17:25:12,972 -  INFO - fouram]: [Base] Params of index: {'index_type': 'IVF_FLAT', 'metric_type': 'L2', 'params': {'nlist': 2048}} (base.py:221)
[2022-10-11 17:25:12,972 -  INFO - fouram]: [CommonCases] Prepare index IVF_FLAT done. (common_cases.py:83)
[2022-10-11 17:25:12,972 -  INFO - fouram]: [Base] Start load collection fouram_5DS5tURl, replica_number:1 (base.py:95)
[2022-10-11 17:27:04,228 -  INFO - fouram]: [Time] Collection.load run in 111.2556s (api_request.py:29)
[2022-10-11 17:27:06,096 -  INFO - fouram]: [PerfTemplate] Actual parameters used: {'collection_params': {'other_fields': []}, 'load_params': {}, 'search_params': {'nq': 1, 'param': {'metric_type': 'L2', 'params': {'nprobe': 8}}, 'top_k': 1, 'expr': None}, 'dataset_params': {'dataset_name': 'sift', 'dim': 128, 'dataset_size': 50000000, 'ni_per': 50000, 'metric_type': 'L2', 'req_run_counts': 30}, 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 2048}}} (performance_template.py:57)
[2022-10-11 17:27:06,097 -  INFO - fouram]: [Base] Params of search: nq:1, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'nprobe': 8}}, limit:1, expr:"None" (base.py:261)
[2022-10-11 17:27:07,522 - ERROR - fouram]: Traceback (most recent call last):
  File "/src/fouram/client/util/api_request.py", line 21, in inner_wrapper
    res = func(*args, **kwargs)
  File "/src/fouram/client/util/api_request.py", line 57, in api_request
    return func(*arg, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 717, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 113, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 139, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 89, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 51, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 472, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 436, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 427, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to Search, QueryNode ID=3, reason=query node 0 is not ready)>
 (api_request.py:35)
[2022-10-11 17:27:07,526 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=fail to Search, QueryNode ID=3, reason=query node 0 is not ready)> (api_request.py:36)
[2022-10-11 17:27:07,526 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=fail to Search, QueryNode ID=3, reason=query node 0 is not ready)> (func_check.py:40)
[2022-10-11 17:27:07,526 - ERROR - fouram]: [Search] Search raise error:  (common_cases.py:401)
[2022-10-11 17:27:07,526 -  INFO - fouram]: [PerfTemplate] Actual parameters used: {'collection_params': {'other_fields': []}, 'load_params': {}, 'search_params': {'nq': 1, 'param': {'metric_type': 'L2', 'params': {'nprobe': 32}}, 'top_k': 1, 'expr': None}, 'dataset_params': {'dataset_name': 'sift', 'dim': 128, 'dataset_size': 50000000, 'ni_per': 50000, 'metric_type': 'L2', 'req_run_counts': 30}, 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 2048}}} (performance_template.py:57)
[2022-10-11 17:27:07,526 -  INFO - fouram]: [Base] Params of search: nq:1, anns_field:float_vector, param:{'metric_type': 'L2', 'params': {'nprobe': 32}}, limit:1, expr:"None" (base.py:261)
[2022-10-11 17:27:07,529 - ERROR - fouram]: Traceback (most recent call last):
  File "/src/fouram/client/util/api_request.py", line 21, in inner_wrapper
    res = func(*args, **kwargs)
  File "/src/fouram/client/util/api_request.py", line 57, in api_request
    return func(*arg, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 717, in search
    res = conn.search(self._name, data, anns_field, param, limit, expr,
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 113, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 109, in handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 139, in handler
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 89, in handler
    raise e
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 51, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 472, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 436, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 427, in _execute_search_requests
    raise MilvusException(response.status.error_code, response.status.reason)
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=3, reason=query node 0 is not ready)>
 (api_request.py:35)
congqixia commented 1 year ago

Fix pr has been merged. Please verify in next runs. Thanks! /unassign /assign @wangting0128

yanliang567 commented 1 year ago

not reproduced, but case failed for #19837