[Bug]: [benchmark][cluster] serach error:status = StatusCode.DEADLINE_EXCEEDED

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications

https://milvus.io

Apache License 2.0

30.81k stars 2.94k forks source link

[Bug]: [benchmark][cluster] serach error:status = StatusCode.DEADLINE_EXCEEDED #11074

Closed wangting0128 closed 2 years ago

wangting0128 commented 3 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):cluster
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus-2.0.0rc8.dev5
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

client.log client log:

[2021-11-01 17:41:29,413] [   DEBUG] - {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 206.0835919380188, 'rps_pv': 20.60835919380188}} (milvus_benchmark.main:106)
[2021-11-01 17:41:29,413] [   DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1635783124, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7f8b2c71fe48>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7f8b2c71ff98>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7f8b2c71ff28>, 'status': 'RUN_SUCC', 'err_message': '', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular'}, 'index': {'index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 256}, 'filter': []}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 206.0835919380188, 'rps_pv': 20.60835919380188}}, 'datetime': '2021-11-01 16:12:04.170266', 'type': 'metric'} (milvus_benchmark.metric.api:29)
[2021-11-01 17:47:25,938] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635788845.935632869","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 17:41:30.778377', 'RPC start': '2021-11-01 17:41:30.778406', 'RPC error': '2021-11-01 17:47:25.937426'} (pymilvus.client.grpc_handler:74)
[2021-11-01 17:47:25,939] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635788845.935632869","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 17:41:29.428535', 'RPC start': '2021-11-01 17:41:29.428546', 'RPC error': '2021-11-01 17:47:25.939911'} (pymilvus.client.grpc_handler:74)
[2021-11-01 17:53:36,367] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635789216.365544722","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 17:47:29.238346', 'RPC start': '2021-11-01 17:47:29.238369', 'RPC error': '2021-11-01 17:53:36.367020'} (pymilvus.client.grpc_handler:74)
[2021-11-01 17:53:36,370] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635789216.365544722","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 17:47:27.961905', 'RPC start': '2021-11-01 17:47:27.961925', 'RPC error': '2021-11-01 17:53:36.370577'} (pymilvus.client.grpc_handler:74)
[2021-11-01 17:59:50,790] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635789590.789495331","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 17:53:39.868450', 'RPC start': '2021-11-01 17:53:39.868481', 'RPC error': '2021-11-01 17:59:50.790263'} (pymilvus.client.grpc_handler:74)
[2021-11-01 17:59:50,802] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635789590.789495331","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 17:53:38.392543', 'RPC start': '2021-11-01 17:53:38.392568', 'RPC error': '2021-11-01 17:59:50.802399'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:06:04,127] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635789964.125503173","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 17:59:54.210832', 'RPC start': '2021-11-01 17:59:54.210862', 'RPC error': '2021-11-01 18:06:04.127156'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:06:05,010] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635789964.125503173","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 17:59:52.822697', 'RPC start': '2021-11-01 17:59:52.822748', 'RPC error': '2021-11-01 18:06:05.010523'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:12:16,516] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635790336.514485274","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:06:08.383060', 'RPC start': '2021-11-01 18:06:08.383098', 'RPC error': '2021-11-01 18:12:16.515835'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:12:16,518] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635790336.514485274","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:06:07.026705', 'RPC start': '2021-11-01 18:06:07.026723', 'RPC error': '2021-11-01 18:12:16.518897'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:18:28,639] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635790708.637580739","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:12:20.126062', 'RPC start': '2021-11-01 18:12:20.126092', 'RPC error': '2021-11-01 18:18:28.638767'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:18:28,651] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635790708.637580739","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:12:18.537479', 'RPC start': '2021-11-01 18:12:18.537504', 'RPC error': '2021-11-01 18:18:28.651655'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:24:41,434] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635791081.432595248","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:18:32.100384', 'RPC start': '2021-11-01 18:18:32.100416', 'RPC error': '2021-11-01 18:24:41.433752'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:24:41,436] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635791081.432595248","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:18:30.668544', 'RPC start': '2021-11-01 18:18:30.668577', 'RPC error': '2021-11-01 18:24:41.436081'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:30:53,463] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635791453.461485233","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:24:44.755649', 'RPC start': '2021-11-01 18:24:44.755685', 'RPC error': '2021-11-01 18:30:53.462774'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:30:53,481] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635791453.461485233","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:24:43.452147', 'RPC start': '2021-11-01 18:24:43.452165', 'RPC error': '2021-11-01 18:30:53.481234'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:37:05,612] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635791825.611632607","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:30:56.890374', 'RPC start': '2021-11-01 18:30:56.890404', 'RPC error': '2021-11-01 18:37:05.612655'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:37:05,620] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635791825.611632607","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:30:55.499339', 'RPC start': '2021-11-01 18:30:55.499357', 'RPC error': '2021-11-01 18:37:05.620251'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:43:21,702] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635792201.700499013","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:37:08.972278', 'RPC start': '2021-11-01 18:37:08.972306', 'RPC error': '2021-11-01 18:43:21.701936'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:43:21,705] [   ERROR] - 
Addr [benchmark-cron-1635782400-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635792201.700499013","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-01 18:37:07.637106', 'RPC start': '2021-11-01 18:37:07.637124', 'RPC error': '2021-11-01 18:43:21.704986'} (pymilvus.client.grpc_handler:74)
[2021-11-01 18:43:21,714] [   ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 45, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1071, in search
    partition_names, output_fields, timeout, round_decimal, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 75, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 57, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 94, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 620, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 75, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 57, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 587, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 575, in _execute_search_requests
    response = ft.result()
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1635792201.700499013","description":"Error received from peer ipv4:10.96.53.235:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 94, in run_suite
    result = runner.run_case(case_metric, **case)
  File "/src/milvus_benchmark/runners/accuracy.py", line 261, in run_case
    self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"])
  File "/src/milvus_benchmark/client.py", line 48, in wrapper
    result = func(*args, **kwargs)
  File "/src/milvus_benchmark/client.py", line 312, in query
    result = self._milvus.search(tmp_collection_name, **params)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 56, in handler
    raise BaseException(1, "rpc timeout")
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>
 (milvus_benchmark.main:97)

server:

NAME                                                              READY   STATUS      RESTARTS   AGE    IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-cron-1635782400-5-etcd-0                                1/1     Running     0          162m   10.97.17.85    qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-etcd-1                                1/1     Running     0          162m   10.97.17.87    qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-etcd-2                                1/1     Running     0          162m   10.97.17.86    qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-milvus-datacoord-74b88fc654-x7qzb     1/1     Running     0          162m   10.97.3.62     qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-milvus-datanode-7f7d747975-9vfs8      1/1     Running     0          162m   10.97.13.43    qa-node010.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-milvus-indexcoord-6f6b5dfcb5-6m7cs    1/1     Running     0          162m   10.97.3.65     qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-milvus-indexnode-5b4cd8df64-92b6s     1/1     Running     0          162m   10.97.12.220   qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-milvus-proxy-7d7645477b-4vtnw         1/1     Running     0          162m   10.97.3.63     qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-milvus-querycoord-57df849dc4-gkb98    1/1     Running     0          162m   10.97.3.64     qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-milvus-querynode-5f9694b578-5kl7k     1/1     Running     0          162m   10.97.15.230   qa-node012.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-milvus-rootcoord-8dcbd9bf8-kdb2x      1/1     Running     0          162m   10.97.8.61     qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-minio-0                               1/1     Running     0          162m   10.97.8.62     qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-minio-1                               1/1     Running     0          162m   10.97.13.42    qa-node010.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-minio-2                               1/1     Running     0          162m   10.97.13.44    qa-node010.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-minio-3                               1/1     Running     0          162m   10.97.6.96     qa-node004.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-pulsar-autorecovery-7f8585874dg98lg   1/1     Running     0          162m   10.97.8.63     qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-pulsar-bastion-7d76d858b4-shcdb       1/1     Running     0          162m   10.97.6.97     qa-node004.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-pulsar-bookkeeper-0                   1/1     Running     0          162m   10.97.6.98     qa-node004.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-pulsar-bookkeeper-1                   1/1     Running     0          161m   10.97.4.103    qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-pulsar-broker-56b54546c7-ldvc6        1/1     Running     0          162m   10.97.7.26     qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-pulsar-proxy-c786c59f7-gl9lc          2/2     Running     0          162m   10.97.9.75     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-pulsar-zookeeper-0                    1/1     Running     0          162m   10.97.8.64     qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-pulsar-zookeeper-1                    1/1     Running     0          162m   10.97.6.99     qa-node004.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-pulsar-zookeeper-2                    1/1     Running     0          162m   10.97.8.66     qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1635782400-5-pulsar-zookeeper-metadata-q7m4b       0/1     Completed   0          162m   10.97.9.74     qa-node007.zilliz.local   <none>           <none>

benchmark-cron-1635782400-5-milvus-indexcoord-6f6b5dfcb5-6m7cs.log

Expected Behavior

No response

Steps To Reproduce

argo task: benchmark-cron-1635782400

test yaml:
client-configmap:client-acc-glove-ivf-flat
server-configmap:server-cluster-8c16m

client pod:benchmark-cron-1635782400-3461446735

Anything else?

No response

binbinlv commented 2 years ago

@czs007 any updates?

wangting0128 commented 2 years ago

argo task:benchmark-cron-1637769600

test yaml: client-configmap:c lient-acc-glove-ivf-flat server-configmap: server-cluster-8c16m

server:

NAME                                                              READY   STATUS      RESTARTS   AGE     IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-cron-1637769600-5-etcd-0                                1/1     Running     0          2m54s   10.97.17.178   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-etcd-1                                1/1     Running     0          2m54s   10.97.17.181   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-etcd-2                                1/1     Running     0          2m54s   10.97.17.184   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-milvus-datacoord-6ccddcd855-lbsz8     1/1     Running     0          2m53s   10.97.9.5      qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-milvus-datanode-85644c9c58-kkx2b      1/1     Running     0          2m52s   10.97.11.154   qa-node009.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-milvus-indexcoord-8b874b78d-d2jsz     1/1     Running     0          2m53s   10.97.9.7      qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-milvus-indexnode-844b457885-9b4hx     1/1     Running     0          2m54s   10.97.7.130    qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-milvus-proxy-f5f455c75-bjbtd          1/1     Running     0          2m54s   10.97.9.6      qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-milvus-querycoord-f98cc666c-xq2zv     1/1     Running     0          2m54s   10.97.9.4      qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-milvus-querynode-744945d4c5-ghmqj     1/1     Running     0          2m52s   10.97.15.24    qa-node012.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-milvus-rootcoord-9f47bcc6c-29p2r      1/1     Running     0          2m53s   10.97.11.153   qa-node009.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-minio-0                               1/1     Running     0          2m54s   10.97.7.128    qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-minio-1                               1/1     Running     0          2m54s   10.97.7.129    qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-minio-2                               1/1     Running     0          2m54s   10.97.8.204    qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-minio-3                               1/1     Running     0          2m54s   10.97.4.27     qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-pulsar-autorecovery-855c5b8f74jfht6   1/1     Running     0          2m53s   10.97.8.205    qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-pulsar-bastion-6448585544-j9gn8       1/1     Running     0          2m52s   10.97.8.206    qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-pulsar-bookkeeper-0                   1/1     Running     0          2m54s   10.97.4.31     qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-pulsar-bookkeeper-1                   1/1     Running     0          75s     10.97.6.57     qa-node004.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-pulsar-broker-5b774df7c8-sl4c2        1/1     Running     0          2m53s   10.97.6.54     qa-node004.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-pulsar-proxy-64cccb6b88-5gpsj         2/2     Running     0          2m53s   10.97.4.28     qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-pulsar-zookeeper-0                    1/1     Running     0          2m54s   10.97.4.29     qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-pulsar-zookeeper-1                    1/1     Running     0          2m35s   10.97.4.32     qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-pulsar-zookeeper-2                    1/1     Running     0          2m13s   10.97.4.33     qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1637769600-5-pulsar-zookeeper-metadata-lzf29       0/1     Completed   0          2m54s   10.97.6.52     qa-node004.zilliz.local   <none>           <none>

client:

[2021-11-24 17:42:48,564] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637775768.560238236","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 17:36:47.947350', 'RPC start': '2021-11-24 17:36:47.947382', 'RPC error': '2021-11-24 17:42:48.564100'} (pymilvus.client.grpc_handler:84)
[2021-11-24 17:42:48,566] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637775768.560238236","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 17:36:46.018599', 'RPC start': '2021-11-24 17:36:46.018629', 'RPC error': '2021-11-24 17:42:48.565900'} (pymilvus.client.grpc_handler:84)
[2021-11-24 17:49:37,995] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637776177.992221515","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 17:42:52.459157', 'RPC start': '2021-11-24 17:42:52.459200', 'RPC error': '2021-11-24 17:49:37.994856'} (pymilvus.client.grpc_handler:84)
[2021-11-24 17:49:38,076] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637776177.992221515","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 17:42:50.589897', 'RPC start': '2021-11-24 17:42:50.589926', 'RPC error': '2021-11-24 17:49:38.075992'} (pymilvus.client.grpc_handler:84)
[2021-11-24 17:56:20,603] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637776580.601278294","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 17:49:41.881654', 'RPC start': '2021-11-24 17:49:41.881699', 'RPC error': '2021-11-24 17:56:20.602961'} (pymilvus.client.grpc_handler:84)
[2021-11-24 17:56:20,608] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637776580.601278294","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 17:49:40.105170', 'RPC start': '2021-11-24 17:49:40.105189', 'RPC error': '2021-11-24 17:56:20.608076'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:03:04,599] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637776984.596297798","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 17:56:24.238898', 'RPC start': '2021-11-24 17:56:24.238933', 'RPC error': '2021-11-24 18:03:04.599189'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:03:05,480] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637776984.596297798","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 17:56:22.631722', 'RPC start': '2021-11-24 17:56:22.631747', 'RPC error': '2021-11-24 18:03:05.480666'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:09:51,987] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637777391.978285645","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:03:09.184188', 'RPC start': '2021-11-24 18:03:09.184235', 'RPC error': '2021-11-24 18:09:51.986461'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:09:52,074] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637777391.978285645","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:03:07.505761', 'RPC start': '2021-11-24 18:03:07.505787', 'RPC error': '2021-11-24 18:09:52.074214'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:16:59,987] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637777819.979183504","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:09:55.801256', 'RPC start': '2021-11-24 18:09:55.801298', 'RPC error': '2021-11-24 18:16:59.987115'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:16:59,992] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637777819.979183504","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:09:54.096606', 'RPC start': '2021-11-24 18:09:54.096627', 'RPC error': '2021-11-24 18:16:59.992459'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:24:04,025] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637778244.020206946","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:17:03.634036', 'RPC start': '2021-11-24 18:17:03.634084', 'RPC error': '2021-11-24 18:24:04.024853'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:24:04,027] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637778244.020206946","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:17:02.017031', 'RPC start': '2021-11-24 18:17:02.017063', 'RPC error': '2021-11-24 18:24:04.027574'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:30:51,983] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637778651.980247174","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:24:07.513918', 'RPC start': '2021-11-24 18:24:07.513952', 'RPC error': '2021-11-24 18:30:51.982842'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:30:51,985] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637778651.980247174","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:24:06.044632', 'RPC start': '2021-11-24 18:24:06.044653', 'RPC error': '2021-11-24 18:30:51.984944'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:37:41,579] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637779061.572551188","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:30:55.757005', 'RPC start': '2021-11-24 18:30:55.757036', 'RPC error': '2021-11-24 18:37:41.578820'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:37:41,581] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637779061.572551188","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:30:54.037931', 'RPC start': '2021-11-24 18:30:54.037950', 'RPC error': '2021-11-24 18:37:41.581755'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:44:43,320] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637779483.298321666","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:37:45.671862', 'RPC start': '2021-11-24 18:37:45.671911', 'RPC error': '2021-11-24 18:44:43.319612'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:44:43,323] [   ERROR] - 
Addr [benchmark-cron-1637769600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637779483.298321666","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-11-24 18:37:43.608531', 'RPC start': '2021-11-24 18:37:43.608581', 'RPC error': '2021-11-24 18:44:43.323131'} (pymilvus.client.grpc_handler:84)
[2021-11-24 18:44:43,333] [   ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1047, in search
    partition_names, output_fields, timeout, round_decimal, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 628, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 595, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 583, in _execute_search_requests
    response = ft.result()
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1637779483.298321666","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 94, in run_suite
    result = runner.run_case(case_metric, **case)
  File "/src/milvus_benchmark/runners/accuracy.py", line 266, in run_case
    self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"])
  File "/src/milvus_benchmark/client.py", line 48, in wrapper
    result = func(*args, **kwargs)
  File "/src/milvus_benchmark/client.py", line 317, in query
    result = self._milvus.search(tmp_collection_name, **params)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 53, in handler
    raise BaseException(1, "rpc timeout")
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>
 (milvus_benchmark.main:97)
[2021-11-24 18:44:43,334] [    INFO] - None (milvus_benchmark.main:98)
[2021-11-24 18:44:43,335] [   DEBUG] - {'type': 'ann_accuracy', 'value': {}} (milvus_benchmark.main:106)
[2021-11-24 18:44:43,336] [   DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1637769927, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7eff1408fe48>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7eff1408fe80>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7eff1408fcc0>, 'status': 'RUN_FAILED', 'err_message': '<BaseException: (code=1, message=rpc timeout)>\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1047, in search\n    partition_names, output_fields, timeout, round_decimal, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler\n    raise e\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 628, in search\n    return self._execute_search_requests(requests, timeout, **_kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler\n    raise e\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 595, in _execute_search_requests\n    raise pre_err\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 583, in _execute_search_requests\n    response = ft.result()\n  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result\n    raise self\ngrpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:\n\tstatus = StatusCode.DEADLINE_EXCEEDED\n\tdetails = "Deadline Exceeded"\n\tdebug_error_string = "{"created":"@1637779483.298321666","description":"Error received from peer ipv4:10.96.159.199:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"\n>\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "main.py", line 94, in run_suite\n    result = runner.run_case(case_metric, **case)\n  File "/src/milvus_benchmark/runners/accuracy.py", line 266, in run_case\n    self.milvus.query(case_param["vector_query"], filter_query=case_param["filter_query"])\n  File "/src/milvus_benchmark/client.py", line 48, in wrapper\n    result = func(*args, **kwargs)\n  File "/src/milvus_benchmark/client.py", line 317, in query\n    result = self._milvus.search(tmp_collection_name, **params)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 53, in handler\n    raise BaseException(1, "rpc timeout")\npymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>\n', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {'index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 512}, 'filter': []}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {}}, 'datetime': '2021-11-24 16:05:27.072972', 'type': 'metric'} (milvus_benchmark.metric.api:29)
[2021-11-24 18:44:43,366] [   DEBUG] - {'_version': '0.1', '_type': 'metric', 'run_id': 1637769927, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7eff1595db00>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7eff1595dd30>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7eff1595d4a8>, 'status': 'RUN_FAILED', 'err_message': '', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {}, 'search': None, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {}}, 'datetime': '2021-11-24 16:05:27.072972', 'type': 'metric'} (milvus_benchmark.metric.api:29)

wangting0128 commented 2 years ago

client-acc-glove-ivf-flat：

          source_file: /test/milvus/ann_hdf5/glove-200-angular.hdf5
          collection_name: glove_200_angular
          index_types: ['ivf_flat']
          index_params:
            nlist: [1024]
          top_ks: [10]
          nqs: [10000]
          search_params:
            nprobe: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]

xiaocai2333 commented 2 years ago

It seems that the historial search took a long time, caused the next search request to timeout. The log:

[2021/11/24 18:41:20.405 +00:00] [DEBUG] [time_recorder.go:76] ["search 429331564171824449(nq=1310, k=10): historical search done (96843ms)"]

xiaocai2333 commented 2 years ago

testing scenarios：

data rows: 1183514
index: IVF_FLAT, nilst: 1024

nq [10000]
nprobe: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
topK: [10]

xiaocai2333 commented 2 years ago

Querynode may be assigned to a machine with poor CPU configuration. It seems that the search time is twice as long as the normal test.

https://argo-test.zilliz.cc/workflows/qa/benchmark-cron-1637856000?tab=workflow&nodeId=benchmark-cron-1637856000-2842082113&nodePanelView=containers&sidePanel=logs:benchmark-cron-1637856000-2842082113

nprobe = 256, same search params： normal test log：

[2021-11-25 17:18:29,338] [   DEBUG] - Milvus query run in 96.99s (milvus_benchmark.client:52)
[2021-11-25 17:20:05,937] [   DEBUG] - Milvus query run in 96.59s (milvus_benchmark.client:52)
[2021-11-25 17:21:57,404] [   DEBUG] - Milvus query run in 111.46s (milvus_benchmark.client:52)
[2021-11-25 17:23:34,183] [   DEBUG] - Milvus query run in 96.77s (milvus_benchmark.client:52)
[2021-11-25 17:25:11,741] [   DEBUG] - Milvus query run in 97.55s (milvus_benchmark.client:52)
[2021-11-25 17:26:49,446] [   DEBUG] - Milvus query run in 97.7s (milvus_benchmark.client:52)
[2021-11-25 17:28:24,948] [   DEBUG] - Milvus query run in 95.44s (milvus_benchmark.client:52)
[2021-11-25 17:28:26,030] [    INFO] - {'acc': 0.979, 'search_rps': 95.44270157814026, 'rps_pv': 9.544270157814026} (milvus_benchmar
k.main:98)
[2021-11-25 17:28:26,031] [   DEBUG] - {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 95.44270157814026, 'rps_pv': 9
.544270157814026}} (milvus_benchmark.main:106)
[2021-11-25 17:28:26,032] [   DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1637856311, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7fee2012a208>, 'hardware': 
<milvus_benchmark.metrics.models.hardware.Hardware object at 0x7fee2012a0b8>, 'env': 
<milvus_benchmark.metrics.models.env.Env object at 0x7fee2012a320>, 'status': 'RUN_SUCC', 'err_message': '', 
'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {'
index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 256}, 
'filter': []}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 95.44270157814026,
'rps_pv':9.544270157814026}}, 'datetime': '2021-11-25 16:05:11.488204', 'type': 'metric'}(milvus_benchmark.metric.api:29)

the test log:

[2021-11-24 17:26:29,285] [   DEBUG] - Milvus query run in 216.82s (milvus_benchmark.client:52)
[2021-11-24 17:29:56,309] [   DEBUG] - Milvus query run in 207.01s (milvus_benchmark.client:52)
[2021-11-24 17:33:19,618] [   DEBUG] - Milvus query run in 203.3s (milvus_benchmark.client:52)
[2021-11-24 17:36:44,589] [   DEBUG] - Milvus query run in 204.88s (milvus_benchmark.client:52)
[2021-11-24 17:36:45,993] [    INFO] - {'acc': 0.979, 'search_rps': 204.8806722164154, 'rps_pv': 20.48806722164154} (milvus_benchmark.main:98)
[2021-11-24 17:36:45,994] [   DEBUG] - {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 204.8806722164154, 'rps_pv': 20.48806722164154}} (milvus_benchmark.main:106)
[2021-11-24 17:36:45,995] [   DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1637769927, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7eff14095208>, 'hardware': 
<milvus_benchmark.metrics.models.hardware.Hardware object at 0x7eff140950b8>, 'env': 
<milvus_benchmark.metrics.models.env.Env object at 0x7eff14095320>, 'status': 'RUN_SUCC', 'err_message': '', 
'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': 
{'index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 256}, 
'filter': []}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 204.8806722164154, 
'rps_pv': 20.48806722164154}}, 'datetime': '2021-11-24 16:05:27.072972', 'type': 'metric'(milvus_benchmark.metric.api:29)

xiaocai2333 commented 2 years ago

Querynode may be assigned to a machine with poor CPU configuration. It seems that the search time is twice as long as the normal test.

nprobe = 256, same search params： normal test log：

[2021-11-25 17:18:29,338] [   DEBUG] - Milvus query run in 96.99s (milvus_benchmark.client:52)
[2021-11-25 17:20:05,937] [   DEBUG] - Milvus query run in 96.59s (milvus_benchmark.client:52)
[2021-11-25 17:21:57,404] [   DEBUG] - Milvus query run in 111.46s (milvus_benchmark.client:52)
[2021-11-25 17:23:34,183] [   DEBUG] - Milvus query run in 96.77s (milvus_benchmark.client:52)
[2021-11-25 17:25:11,741] [   DEBUG] - Milvus query run in 97.55s (milvus_benchmark.client:52)
[2021-11-25 17:26:49,446] [   DEBUG] - Milvus query run in 97.7s (milvus_benchmark.client:52)
[2021-11-25 17:28:24,948] [   DEBUG] - Milvus query run in 95.44s (milvus_benchmark.client:52)
[2021-11-25 17:28:26,030] [    INFO] - {'acc': 0.979, 'search_rps': 95.44270157814026, 'rps_pv': 9.544270157814026} (milvus_benchmar
k.main:98)
[2021-11-25 17:28:26,031] [   DEBUG] - {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 95.44270157814026, 'rps_pv': 9
.544270157814026}} (milvus_benchmark.main:106)
[2021-11-25 17:28:26,032] [   DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1637856311, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7fee2012a208>, 'hardware': 
<milvus_benchmark.metrics.models.hardware.Hardware object at 0x7fee2012a0b8>, 'env': 
<milvus_benchmark.metrics.models.env.Env object at 0x7fee2012a320>, 'status': 'RUN_SUCC', 'err_message': '', 
'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {'
index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 256}, 
'filter': []}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 95.44270157814026,
'rps_pv':9.544270157814026}}, 'datetime': '2021-11-25 16:05:11.488204', 'type': 'metric'}(milvus_benchmark.metric.api:29)

the test log:

[2021-11-24 17:26:29,285] [   DEBUG] - Milvus query run in 216.82s (milvus_benchmark.client:52)
[2021-11-24 17:29:56,309] [   DEBUG] - Milvus query run in 207.01s (milvus_benchmark.client:52)
[2021-11-24 17:33:19,618] [   DEBUG] - Milvus query run in 203.3s (milvus_benchmark.client:52)
[2021-11-24 17:36:44,589] [   DEBUG] - Milvus query run in 204.88s (milvus_benchmark.client:52)
[2021-11-24 17:36:45,993] [    INFO] - {'acc': 0.979, 'search_rps': 204.8806722164154, 'rps_pv': 20.48806722164154} (milvus_benchmark.main:98)
[2021-11-24 17:36:45,994] [   DEBUG] - {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 204.8806722164154, 'rps_pv': 20.48806722164154}} (milvus_benchmark.main:106)
[2021-11-24 17:36:45,995] [   DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1637769927, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7eff14095208>, 'hardware': 
<milvus_benchmark.metrics.models.hardware.Hardware object at 0x7eff140950b8>, 'env': 
<milvus_benchmark.metrics.models.env.Env object at 0x7eff14095320>, 'status': 'RUN_SUCC', 'err_message': '', 
'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': 
{'index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 256}, 
'filter': []}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {'acc': 0.979, 'search_rps': 204.8806722164154, 
'rps_pv': 20.48806722164154}}, 'datetime': '2021-11-24 16:05:27.072972', 'type': 'metric'(milvus_benchmark.metric.api:29)

Maybe this is a wrong understanding, pod will apply the specified cpu resources when being scheduled.

jingkl commented 2 years ago

client-configmap：client-search-sift50m-ivf-flat-1024, server-configmap:server-cluster-8c32m argo task：benchmark-tag-jx52w sever ：

NAME                                                         READY   STATUS      RESTARTS   AGE     IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-tag-jx52w-1-milvus-datanode-bc5b4d469-77fnk        1/1     Running     0          3m58s   10.97.20.197   qa-node018.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-milvus-indexcoord-5fdf7988b9-fqr9b     1/1     Running     0          3m58s   10.97.7.152    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-milvus-indexnode-587b8ccdf8-xlmrb      1/1     Running     0          3m58s   10.97.14.122   qa-node011.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-milvus-proxy-b68d8c7cb-fdbmz           1/1     Running     0          3m58s   10.97.6.184    qa-node004.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-milvus-querycoord-5d7b574dc4-zvcvp     1/1     Running     0          3m58s   10.97.6.183    qa-node004.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-milvus-querynode-58c888f7bd-7l796      1/1     Running     0          3m58s   10.97.16.21    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-milvus-rootcoord-d95fd4b94-6ddfd       1/1     Running     0          3m58s   10.97.6.185    qa-node004.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-minio-0                                1/1     Running     0          3m58s   10.97.12.170   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-minio-1                                1/1     Running     0          3m58s   10.97.12.169   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-minio-2                                1/1     Running     0          3m58s   10.97.12.168   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-minio-3                                1/1     Running     0          3m58s   10.97.12.171   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-autorecovery-6944d858d-mh4wz    1/1     Running     0          3m58s   10.97.9.246    qa-node007.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-bastion-5d94c49d84-cbm4h        1/1     Running     0          3m58s   10.97.7.154    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-bookkeeper-0                    1/1     Running     0          3m58s   10.97.11.143   qa-node009.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-bookkeeper-1                    1/1     Running     0          2m23s   10.97.13.51    qa-node010.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-bookkeeper-2                    1/1     Running     0          95s     10.97.11.144   qa-node009.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-broker-66bcfdc98d-2gnhg         1/1     Running     0          3m58s   10.97.8.89     qa-node006.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-proxy-886f47996-45f7g           2/2     Running     0          3m58s   10.97.10.125   qa-node008.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-zookeeper-0                     1/1     Running     0          3m58s   10.97.7.155    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-zookeeper-1                     1/1     Running     0          3m35s   10.97.7.156    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-zookeeper-2                     1/1     Running     0          3m14s   10.97.3.149    qa-node001.zilliz.local   <none>           <none>
benchmark-tag-jx52w-1-pulsar-zookeeper-metadata-4pgdv        0/1     Completed   0          3m58s   10.97.8.90     qa-node006.zilliz.local   <none>           <none>

client log：

2021-12-06 16:08:37,242] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638806917.240217744","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:03:37.239124', 'RPC start': '2021-12-06 16:03:37.239146', 'RPC error': '2021-12-06 16:08:37.242319'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:08:37,248] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638806917.240217744","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:03:37.169269', 'RPC start': '2021-12-06 16:03:37.169284', 'RPC error': '2021-12-06 16:08:37.248632'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:18:38,325] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638807518.321304008","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:13:38.320712', 'RPC start': '2021-12-06 16:13:38.320731', 'RPC error': '2021-12-06 16:18:38.324939'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:18:38,340] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638807518.321304008","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:13:38.253826', 'RPC start': '2021-12-06 16:13:38.253851', 'RPC error': '2021-12-06 16:18:38.340144'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:28:39,414] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638808119.410284895","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:23:39.409132', 'RPC start': '2021-12-06 16:23:39.409154', 'RPC error': '2021-12-06 16:28:39.414222'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:28:39,432] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638808119.410284895","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:23:39.337129', 'RPC start': '2021-12-06 16:23:39.337166', 'RPC error': '2021-12-06 16:28:39.432048'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:38:40,500] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638808720.496345456","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:33:40.495249', 'RPC start': '2021-12-06 16:33:40.495272', 'RPC error': '2021-12-06 16:38:40.500174'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:38:40,502] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638808720.496345456","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:33:40.425795', 'RPC start': '2021-12-06 16:33:40.425821', 'RPC error': '2021-12-06 16:38:40.502007'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:48:41,585] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638809321.583208161","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:43:41.582012', 'RPC start': '2021-12-06 16:43:41.582033', 'RPC error': '2021-12-06 16:48:41.585563'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:48:41,587] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638809321.583208161","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:43:41.511944', 'RPC start': '2021-12-06 16:43:41.511978', 'RPC error': '2021-12-06 16:48:41.587614'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:58:42,663] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638809922.661256497","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:53:42.660583', 'RPC start': '2021-12-06 16:53:42.660605', 'RPC error': '2021-12-06 16:58:42.663433'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:58:42,732] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638809922.661256497","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:53:42.594792', 'RPC start': '2021-12-06 16:53:42.594827', 'RPC error': '2021-12-06 16:58:42.731949'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:08:43,739] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638810523.736264869","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:03:43.735313', 'RPC start': '2021-12-06 17:03:43.735334', 'RPC error': '2021-12-06 17:08:43.738761'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:08:43,740] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638810523.736264869","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:03:43.671698', 'RPC start': '2021-12-06 17:03:43.671721', 'RPC error': '2021-12-06 17:08:43.740505'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:18:44,844] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638811124.841296612","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:13:44.840250', 'RPC start': '2021-12-06 17:13:44.840269', 'RPC error': '2021-12-06 17:18:44.844074'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:18:44,847] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638811124.841296612","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:13:44.769312', 'RPC start': '2021-12-06 17:13:44.769336', 'RPC error': '2021-12-06 17:18:44.847015'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:28:45,937] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638811725.931249822","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:23:45.930790', 'RPC start': '2021-12-06 17:23:45.930811', 'RPC error': '2021-12-06 17:28:45.936922'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:28:45,966] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638811725.931249822","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:23:45.854845', 'RPC start': '2021-12-06 17:23:45.854869', 'RPC error': '2021-12-06 17:28:45.966162'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:38:47,027] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638812327.024267429","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:33:47.023662', 'RPC start': '2021-12-06 17:33:47.023683', 'RPC error': '2021-12-06 17:38:47.027612'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:38:47,029] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638812327.024267429","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:33:46.946570', 'RPC start': '2021-12-06 17:33:46.946594', 'RPC error': '2021-12-06 17:38:47.029578'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:38:47,032] [   ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1048, in search
    partition_names, output_fields, timeout, round_decimal, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 643, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 605, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 593, in _execute_search_requests
    response = ft.result()
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638812327.024267429","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

During handling of the above exception, another exception occurred:
2021-12-06 16:08:37,242] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638806917.240217744","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:03:37.239124', 'RPC start': '2021-12-06 16:03:37.239146', 'RPC error': '2021-12-06 16:08:37.242319'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:08:37,248] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638806917.240217744","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:03:37.169269', 'RPC start': '2021-12-06 16:03:37.169284', 'RPC error': '2021-12-06 16:08:37.248632'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:18:38,325] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638807518.321304008","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:13:38.320712', 'RPC start': '2021-12-06 16:13:38.320731', 'RPC error': '2021-12-06 16:18:38.324939'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:18:38,340] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638807518.321304008","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:13:38.253826', 'RPC start': '2021-12-06 16:13:38.253851', 'RPC error': '2021-12-06 16:18:38.340144'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:28:39,414] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638808119.410284895","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:23:39.409132', 'RPC start': '2021-12-06 16:23:39.409154', 'RPC error': '2021-12-06 16:28:39.414222'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:28:39,432] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638808119.410284895","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:23:39.337129', 'RPC start': '2021-12-06 16:23:39.337166', 'RPC error': '2021-12-06 16:28:39.432048'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:38:40,500] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638808720.496345456","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:33:40.495249', 'RPC start': '2021-12-06 16:33:40.495272', 'RPC error': '2021-12-06 16:38:40.500174'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:38:40,502] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638808720.496345456","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:33:40.425795', 'RPC start': '2021-12-06 16:33:40.425821', 'RPC error': '2021-12-06 16:38:40.502007'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:48:41,585] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638809321.583208161","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:43:41.582012', 'RPC start': '2021-12-06 16:43:41.582033', 'RPC error': '2021-12-06 16:48:41.585563'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:48:41,587] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638809321.583208161","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:43:41.511944', 'RPC start': '2021-12-06 16:43:41.511978', 'RPC error': '2021-12-06 16:48:41.587614'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:58:42,663] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638809922.661256497","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:53:42.660583', 'RPC start': '2021-12-06 16:53:42.660605', 'RPC error': '2021-12-06 16:58:42.663433'} (pymilvus.client.grpc_handler:84)
[2021-12-06 16:58:42,732] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638809922.661256497","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 16:53:42.594792', 'RPC start': '2021-12-06 16:53:42.594827', 'RPC error': '2021-12-06 16:58:42.731949'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:08:43,739] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638810523.736264869","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:03:43.735313', 'RPC start': '2021-12-06 17:03:43.735334', 'RPC error': '2021-12-06 17:08:43.738761'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:08:43,740] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638810523.736264869","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:03:43.671698', 'RPC start': '2021-12-06 17:03:43.671721', 'RPC error': '2021-12-06 17:08:43.740505'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:18:44,844] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638811124.841296612","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:13:44.840250', 'RPC start': '2021-12-06 17:13:44.840269', 'RPC error': '2021-12-06 17:18:44.844074'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:18:44,847] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638811124.841296612","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:13:44.769312', 'RPC start': '2021-12-06 17:13:44.769336', 'RPC error': '2021-12-06 17:18:44.847015'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:28:45,937] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638811725.931249822","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:23:45.930790', 'RPC start': '2021-12-06 17:23:45.930811', 'RPC error': '2021-12-06 17:28:45.936922'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:28:45,966] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638811725.931249822","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:23:45.854845', 'RPC start': '2021-12-06 17:23:45.854869', 'RPC error': '2021-12-06 17:28:45.966162'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:38:47,027] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638812327.024267429","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:33:47.023662', 'RPC start': '2021-12-06 17:33:47.023683', 'RPC error': '2021-12-06 17:38:47.027612'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:38:47,029] [   ERROR] - 
Addr [benchmark-tag-jx52w-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638812327.024267429","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 17:33:46.946570', 'RPC start': '2021-12-06 17:33:46.946594', 'RPC error': '2021-12-06 17:38:47.029578'} (pymilvus.client.grpc_handler:84)
[2021-12-06 17:38:47,032] [   ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1048, in search
    partition_names, output_fields, timeout, round_decimal, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 643, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 605, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 593, in _execute_search_requests
    response = ft.result()
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638812327.024267429","description":"Error received from peer ipv4:10.96.165.243:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

During handling of the above exception, another exception occurred:

jingkl commented 2 years ago

client-configmap:client-search-sift50m-ivf-flat-1024,i server-configmap:server-cluster-8c32m-segment1024 server：

NAME                                                         READY   STATUS      RESTARTS   AGE     IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-tag-pkbxr-1-etcd-0                                 1/1     Running     0          5m58s   10.97.17.101   qa-node014.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-etcd-1                                 1/1     Running     0          5m58s   10.97.16.17    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-etcd-2                                 1/1     Running     0          5m58s   10.97.16.20    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-milvus-datacoord-68bf8bbfd-w7vq7       1/1     Running     1          5m58s   10.97.7.147    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-milvus-datanode-5c777d6459-bcsc8       1/1     Running     1          5m58s   10.97.19.91    qa-node016.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-milvus-indexcoord-9857db6b7-df66j      1/1     Running     0          5m58s   10.97.7.150    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-milvus-indexnode-68c4845b74-n5bb4      1/1     Running     0          5m58s   10.97.17.99    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-milvus-proxy-66c5f7b4f6-g7dvn          1/1     Running     1          5m58s   10.97.7.146    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-milvus-querycoord-5ddf5588cb-tc8dv     1/1     Running     1          5m58s   10.97.7.148    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-milvus-querynode-766464df96-24srh      1/1     Running     0          5m58s   10.97.20.196   qa-node018.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-milvus-rootcoord-5ccc9d4f74-xds6c      1/1     Running     0          5m58s   10.97.7.145    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-minio-0                                1/1     Running     0          5m58s   10.97.19.93    qa-node016.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-minio-1                                1/1     Running     0          5m58s   10.97.16.16    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-minio-2                                1/1     Running     0          5m58s   10.97.16.18    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-minio-3                                1/1     Running     0          5m58s   10.97.12.163   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-autorecovery-6c4bc9bd78-sjx5x   1/1     Running     0          5m58s   10.97.11.141   qa-node009.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-bastion-59f9f464fb-q4b52        1/1     Running     0          5m58s   10.97.9.242    qa-node007.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-bookkeeper-0                    1/1     Running     0          5m58s   10.97.10.124   qa-node008.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-bookkeeper-1                    1/1     Running     0          4m11s   10.97.11.142   qa-node009.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-bookkeeper-2                    1/1     Running     0          3m17s   10.97.13.50    qa-node010.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-broker-74c7b9587f-lw68r         1/1     Running     0          5m58s   10.97.3.148    qa-node001.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-proxy-5f69f8455c-wtc57          2/2     Running     0          5m58s   10.97.10.123   qa-node008.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-zookeeper-0                     1/1     Running     0          5m58s   10.97.9.244    qa-node007.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-zookeeper-1                     1/1     Running     0          5m42s   10.97.8.88     qa-node006.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-zookeeper-2                     1/1     Running     0          5m16s   10.97.9.245    qa-node007.zilliz.local   <none>           <none>
benchmark-tag-pkbxr-1-pulsar-zookeeper-metadata-b6xgv        0/1     Completed   0          5m58s   10.97.9.243    qa-node007.zilliz.local   <none>           <none>

client log：

[2021-12-06 15:20:11,199] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638804011.198026629","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 15:15:11.197441', 'RPC start': '2021-12-06 15:15:11.197462', 'RPC error': '2021-12-06 15:20:11.198919'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:20:11,200] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638804011.198026629","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 15:15:11.116482', 'RPC start': '2021-12-06 15:15:11.116500', 'RPC error': '2021-12-06 15:20:11.200396'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:25:13,294] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638804313.293966515","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 15:20:13.293067', 'RPC start': '2021-12-06 15:20:13.293085', 'RPC error': '2021-12-06 15:25:13.294822'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:25:13,296] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638804313.293966515","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 15:20:13.213833', 'RPC start': '2021-12-06 15:20:13.213859', 'RPC error': '2021-12-06 15:25:13.295960'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:30:15,387] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638804615.385956711","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 15:25:15.385377', 'RPC start': '2021-12-06 15:25:15.385396', 'RPC error': '2021-12-06 15:30:15.387564'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:30:15,389] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638804615.385956711","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 15:25:15.308781', 'RPC start': '2021-12-06 15:25:15.308805', 'RPC error': '2021-12-06 15:30:15.389116'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:35:17,495] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638804917.492924438","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 15:30:17.492532', 'RPC start': '2021-12-06 15:30:17.492552', 'RPC error': '2021-12-06 15:35:17.494781'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:35:17,496] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638804917.492924438","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-06 15:30:17.406869', 'RPC start': '2021-12-06 15:30:17.406893', 'RPC error': '2021-12-06 15:35:17.496439'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:40:19,620] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1638805219.618980406","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

client pod：benchmark-tag-pkbxr-850577231

jingkl commented 2 years ago

op：

1、create collection
2、create index
3、insert 50m vectors
4、flush
5、create index with the same params
6、load collection
7、search <- raise error

xiaocai2333 commented 2 years ago

client-configmap:client-search-sift50m-ivf-flat-1024,i server-configmap:server-cluster-8c32m-segment1024 server：

NAME                                                         READY   STATUS      RESTARTS   AGE     IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-tag-g9pd2-1-etcd-0                                 1/1     Running     0          5m45s   10.97.17.93    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-etcd-1                                 1/1     Running     0          5m45s   10.97.16.7     qa-node013.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-etcd-2                                 1/1     Running     0          5m45s   10.97.17.95    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-milvus-datacoord-59f95b498b-8m2sk      1/1     Running     1          5m46s   10.97.6.180    qa-node004.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-milvus-datanode-77f9fb97d9-flwk7       1/1     Running     1          5m45s   10.97.19.86    qa-node016.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-milvus-indexcoord-65d4c89669-2px2m     1/1     Running     0          5m45s   10.97.7.132    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-milvus-indexnode-67856cf74b-jwd8f      1/1     Running     0          5m46s   10.97.6.182    qa-node004.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-milvus-proxy-7c67ffc95b-qdsm5          1/1     Running     1          5m45s   10.97.7.131    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-milvus-querycoord-56fb5bb6c4-w5t2v     1/1     Running     1          5m45s   10.97.19.85    qa-node016.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-milvus-querynode-7b78fc776-99ck6       1/1     Running     1          5m45s   10.97.20.194   qa-node018.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-milvus-rootcoord-568fcb5c78-jlqgj      1/1     Running     1          5m46s   10.97.6.181    qa-node004.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-minio-0                                1/1     Running     0          5m46s   10.97.19.88    qa-node016.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-minio-1                                1/1     Running     0          5m45s   10.97.12.150   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-minio-2                                1/1     Running     0          5m45s   10.97.12.152   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-minio-3                                1/1     Running     0          5m45s   10.97.12.153   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-autorecovery-648c856864-txjwx   1/1     Running     0          5m46s   10.97.10.110   qa-node008.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-bastion-889b65dd8-lgrfn         1/1     Running     0          5m45s   10.97.9.220    qa-node007.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-bookkeeper-0                    1/1     Running     0          5m45s   10.97.7.133    qa-node005.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-bookkeeper-1                    1/1     Running     0          4m1s    10.97.11.137   qa-node009.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-bookkeeper-2                    1/1     Running     0          3m4s    10.97.10.116   qa-node008.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-broker-768f474bcb-z2dqx         1/1     Running     0          5m46s   10.97.8.81     qa-node006.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-proxy-5fd46f9648-4t8bw          2/2     Running     0          5m45s   10.97.13.43    qa-node010.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-zookeeper-0                     1/1     Running     0          5m45s   10.97.9.221    qa-node007.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-zookeeper-1                     1/1     Running     0          5m12s   10.97.13.44    qa-node010.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-zookeeper-2                     1/1     Running     0          4m52s   10.97.10.115   qa-node008.zilliz.local   <none>           <none>
benchmark-tag-g9pd2-1-pulsar-zookeeper-metadata-ppxxz        0/1     Completed   0          5m46s   10.97.7.130    qa-node005.zilliz.local   <none>           <none>

client log：

[2021-12-06 15:20:11,199] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
  status = StatusCode.DEADLINE_EXCEEDED
  details = "Deadline Exceeded"
  debug_error_string = "{"created":"@1638804011.198026629","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
  {'API start': '2021-12-06 15:15:11.197441', 'RPC start': '2021-12-06 15:15:11.197462', 'RPC error': '2021-12-06 15:20:11.198919'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:20:11,200] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
  status = StatusCode.DEADLINE_EXCEEDED
  details = "Deadline Exceeded"
  debug_error_string = "{"created":"@1638804011.198026629","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
  {'API start': '2021-12-06 15:15:11.116482', 'RPC start': '2021-12-06 15:15:11.116500', 'RPC error': '2021-12-06 15:20:11.200396'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:25:13,294] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
  status = StatusCode.DEADLINE_EXCEEDED
  details = "Deadline Exceeded"
  debug_error_string = "{"created":"@1638804313.293966515","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
  {'API start': '2021-12-06 15:20:13.293067', 'RPC start': '2021-12-06 15:20:13.293085', 'RPC error': '2021-12-06 15:25:13.294822'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:25:13,296] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
  status = StatusCode.DEADLINE_EXCEEDED
  details = "Deadline Exceeded"
  debug_error_string = "{"created":"@1638804313.293966515","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
  {'API start': '2021-12-06 15:20:13.213833', 'RPC start': '2021-12-06 15:20:13.213859', 'RPC error': '2021-12-06 15:25:13.295960'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:30:15,387] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
  status = StatusCode.DEADLINE_EXCEEDED
  details = "Deadline Exceeded"
  debug_error_string = "{"created":"@1638804615.385956711","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
  {'API start': '2021-12-06 15:25:15.385377', 'RPC start': '2021-12-06 15:25:15.385396', 'RPC error': '2021-12-06 15:30:15.387564'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:30:15,389] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
  status = StatusCode.DEADLINE_EXCEEDED
  details = "Deadline Exceeded"
  debug_error_string = "{"created":"@1638804615.385956711","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
  {'API start': '2021-12-06 15:25:15.308781', 'RPC start': '2021-12-06 15:25:15.308805', 'RPC error': '2021-12-06 15:30:15.389116'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:35:17,495] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
  status = StatusCode.DEADLINE_EXCEEDED
  details = "Deadline Exceeded"
  debug_error_string = "{"created":"@1638804917.492924438","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
  {'API start': '2021-12-06 15:30:17.492532', 'RPC start': '2021-12-06 15:30:17.492552', 'RPC error': '2021-12-06 15:35:17.494781'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:35:17,496] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
  status = StatusCode.DEADLINE_EXCEEDED
  details = "Deadline Exceeded"
  debug_error_string = "{"created":"@1638804917.492924438","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
  {'API start': '2021-12-06 15:30:17.406869', 'RPC start': '2021-12-06 15:30:17.406893', 'RPC error': '2021-12-06 15:35:17.496439'} (pymilvus.client.grpc_handler:84)
[2021-12-06 15:40:19,620] [   ERROR] - 
Addr [benchmark-tag-pkbxr-1-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
  status = StatusCode.DEADLINE_EXCEEDED
  details = "Deadline Exceeded"
  debug_error_string = "{"created":"@1638805219.618980406","description":"Error received from peer ipv4:10.96.174.36:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

client pod：benchmark-tag-pkbxr-850577231

The reason of this problem is: the querynode can't load 50M data.

[2021/12/06 12:40:04.664 +00:00] [WARN] [task.go:729] ["load segment failed, OOM if load, collectionID =  429596664774262785 ,  usedMem =  25939226624 ,  segmentTotalSize =  4157425753 ,  currentSegmentSize =  831595365 ,  totalMem =  34359738368 ,  thresholdFactor =  0.9\n"]

What is the dim of the data?

xiaocai2333 commented 2 years ago

querynode is panic

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

runtime stack:
    /usr/local/go/src/runtime/panic.go:1116 +0x72

github.com/milvus-io/milvus/internal/querynode._Cfunc_ReduceSearchResultsAndFillData(0x7f2794716e40, 0xc001dd8000, 0x22, 0x0, 0x0)
    /go/src/github.com/milvus-io/milvus/internal/querynode/query_collection.go:1041 +0x16da fp=0xc004b8da20 sp=0xc004b8d180 pc=0x18b5cba
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc00091feff, 0x0, 0x0, 0x0)

cydrain commented 2 years ago

querynode is panic

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

runtime stack:
  /usr/local/go/src/runtime/panic.go:1116 +0x72

github.com/milvus-io/milvus/internal/querynode._Cfunc_ReduceSearchResultsAndFillData(0x7f2794716e40, 0xc001dd8000, 0x22, 0x0, 0x0)
  /go/src/github.com/milvus-io/milvus/internal/querynode/query_collection.go:1041 +0x16da fp=0xc004b8da20 sp=0xc004b8d180 pc=0x18b5cba
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc00091feff, 0x0, 0x0, 0x0)

looks like similar as #10530

goroutine 36342 [syscall]:
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7fc4afc4e000 pc=0x7fc4e5784e43]
runtime.throw(0x265ad6e, 0x2a)

/usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:704 +0x4ac
runtime.cgocall(0x17c049a, 0xc00076d038, 0xc02dafcc00)
/usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc00076d008 sp=0xc00076cfd0 pc=0x729d1b
/go/src/github.com/milvus-io/milvus/internal/querynode/reduce.go:49 +0x8f fp=0xc00076d078 sp=0xc00076d038 pc=0x17a844f
github.com/milvus-io/milvus/internal/querynode.reduceSearchResultsAndFillData(0xc0003cb748, 0xc030bf8120, 0x22, 0x24, 0x22, 0x0, 0x0)
/go/src/github.com/milvus-io/milvus/internal/querynode/reduce.go:49 +0x178 fp=0xc00076d130 sp=0xc00076d078 pc=0x179c938
/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc00076dfe0 sp=0xc00076dfd8 pc=0x796641
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc0007ffeff, 0x0, 0x0, 0x0)

cydrain commented 2 years ago

rerun benchmark, but fail to reproduce

https://argo-test.zilliz.cc/workflows/yudong/yudong-benchmark-hl596?tab=workflow&nodeId=yudong-benchmark-hl596-3056709678

cydrain commented 2 years ago

/assign

cydrain commented 2 years ago

https://argo-test.zilliz.cc/workflows/yudong/yudong-benchmark-ll2v5?tab=workflow&nodeId=yudong-benchmark-ll2v5-2479816754

NAME                                                            READY   STATUS      RESTARTS   AGE     IP             NODE                      NOMINATED NODE   READINESS GATES
yudong-benchmark-ll2v5-1-etcd-0                                 1/1     Running     0          3m55s   10.97.16.21    qa-node013.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-etcd-1                                 1/1     Running     0          3m55s   10.97.17.247   qa-node014.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-etcd-2                                 1/1     Running     0          3m54s   10.97.16.22    qa-node013.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-milvus-datacoord-8444b7485c-tclwv      1/1     Running     0          3m54s   10.97.9.221    qa-node007.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-milvus-datanode-79f5b6576-fz27t        1/1     Running     0          3m54s   10.97.10.40    qa-node008.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-milvus-indexcoord-c95869786-gdc9j      1/1     Running     0          3m54s   10.97.7.194    qa-node005.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-milvus-indexnode-6cccb56647-xwf72      1/1     Running     0          3m55s   10.97.17.245   qa-node014.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-milvus-proxy-7c64fc95f-cwtpf           1/1     Running     0          3m55s   10.97.7.193    qa-node005.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-milvus-querycoord-777b4fc5d5-445xr     1/1     Running     0          3m54s   10.97.7.196    qa-node005.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-milvus-querynode-7d7ff95ff7-95v79      1/1     Running     0          3m55s   10.97.14.231   qa-node011.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-milvus-rootcoord-f7676c745-9cdz2       1/1     Running     0          3m54s   10.97.9.220    qa-node007.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-minio-0                                1/1     Running     0          3m55s   10.97.20.244   qa-node018.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-minio-1                                1/1     Running     0          3m55s   10.97.20.245   qa-node018.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-minio-2                                1/1     Running     0          3m55s   10.97.20.246   qa-node018.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-minio-3                                1/1     Running     0          3m54s   10.97.20.248   qa-node018.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-autorecovery-6c8fcf6468-49pn7   1/1     Running     0          3m55s   10.97.4.215    qa-node002.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-bastion-9f4986d74-brrtb         1/1     Running     0          3m55s   10.97.4.216    qa-node002.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-bookkeeper-0                    1/1     Running     0          3m55s   10.97.8.86     qa-node006.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-bookkeeper-1                    1/1     Running     0          2m10s   10.97.4.219    qa-node002.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-bookkeeper-2                    1/1     Running     0          102s    10.97.11.51    qa-node009.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-broker-56d48d54cf-nqwzt         1/1     Running     0          3m54s   10.97.7.195    qa-node005.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-proxy-596c8fbd7f-4qgq2          2/2     Running     0          3m54s   10.97.3.141    qa-node001.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-zookeeper-0                     1/1     Running     0          3m55s   10.97.9.222    qa-node007.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-zookeeper-1                     1/1     Running     0          3m34s   10.97.4.218    qa-node002.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-zookeeper-2                     1/1     Running     0          3m18s   10.97.11.49    qa-node009.zilliz.local   <none>           <none>
yudong-benchmark-ll2v5-1-pulsar-zookeeper-metadata-mrdmh        0/1     Completed   0          3m55s   10.97.3.140    qa-node001.zilliz.local   <none>           <none>

cydrain commented 2 years ago

in 1206 run:

[2021/12/06 15:16:27.645 +00:00] [DEBUG] [time_recorder.go:76] ["search 429597039624129857(nq=1200, k=1): streaming search done (0ms)"]
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

runtime stack:
    /usr/local/go/src/runtime/panic.go:1116 +0x72

github.com/milvus-io/milvus/internal/querynode._Cfunc_ReduceSearchResultsAndFillData(0x7f2794716e40, 0xc001dd8000, 0x22, 0x0, 0x0)
    /go/src/github.com/milvus-io/milvus/internal/querynode/query_collection.go:1041 +0x16da fp=0xc004b8da20 sp=0xc004b8d180 pc=0x18b5cba
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc00091feff, 0x0, 0x0, 0x0)
[2021/12/06 15:16:27.649 +00:00] [DEBUG] [segment.go:232] ["delete segment"] [segmentID=429597472987283457]
[2021/12/06 15:16:27.649 +00:00] [DEBUG] [collection_replica.go:553] ["remove segment"] [segmentID=429597348193370114]
main.main()
    /go/src/github.com/milvus-io/milvus/cmd/main.go:303 +0x649
goroutine 607 [select, 358 minutes]:
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/balancer_conn_wrappers.go:69 +0xc8
    /go/pkg/mod/google.golang.org/grpc@v1.38.0/balancer_conn_wrappers.go:69 +0xc8
goroutine 868 [select]:
internal/poll.runtime_pollWait(0x7f2ca80c2808, 0x72, 0x0)
internal/poll.(*FD).Accept(0xc0002ee500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2937 +0x266
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:704 +0x4ac
    /usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc004b8d098 sp=0xc004b8d060 pc=0x7c8a9b
    _cgo_gotypes.go:723 +0x56 fp=0xc004b8d0c8 sp=0xc004b8d098 pc=0x18d55b6
    /go/src/github.com/milvus-io/milvus/internal/querynode/reduce.go:55 +0x8f fp=0xc004b8d108 sp=0xc004b8d0c8 pc=0x18efe2f
github.com/milvus-io/milvus/internal/querynode.reduceSearchResultsAndFillData(0xc0043a2478, 0xc0016a6d80, 0x22, 0x24, 0x22, 0x0, 0x0)
    /go/src/github.com/milvus-io/milvus/internal/querynode/reduce.go:55 +0x13d fp=0xc004b8d180 sp=0xc004b8d108 pc=0x18e469d
github.com/milvus-io/milvus/internal/querynode.(*queryCollection).doUnsolvedQueryMsg(0xc001772600)
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc004b8dfe0 sp=0xc004b8dfd8 pc=0x835481
goroutine 1 [chan receive, 359 minutes]:
    /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:488 +0x44e

in 1215 run:

[2021/12/14 08:54:52.944 +00:00] [DEBUG] [time_recorder.go:76] ["search 429773294952451393(nq=20, k=1000): streaming search done (1ms)"]
2021-12-14 08:54:52,988 DEBUG [default] [user@yudong-benchmark-ll2v5-1-milvus-querynode-7d7ff95ff7-95v79] [void ReduceResultData(std::vector<milvus::SearchResult*>&, int64_t, int64_t)] [/go/src/github.com/milvus-io/milvus/internal/core/src/segcore/reduce_c.cpp:77] [SEGCORE][ReduceResultData][milvus] reduce search result data, num of segments 66, nq 20, topk 1000
... ...
{ seg No.11 (8000, 49881370, -0.311019), (8001, 49664046, -0.399031), ......, (8563, 49964454, -0.981177), (8564, 49841218,

cydrain commented 2 years ago

For #10530, the root cause is, in segment 7, all data are duplicated with primary key "7737081", the reduce code cannot handle this case. For #11074, the 2 runs panic with different search parameters: 1206 run, nq=1200, k=1 1215 run, nq=20, k=1000, see no abnormal in debug log, seems not reduce error

Also double-checked the code how segment is protected by RWLOCK (#6870), the code looks fine.

czs007 commented 2 years ago

@xiaocai2333 is working on this.

wangting0128 commented 2 years ago

argo task: benchmark-cron-1639843200

test yaml: client-configmap:client-acc-glove-ivf-flat server-configmap:server-cluster-8c16m

server:

NAME                                                              READY   STATUS             RESTARTS   AGE     IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-cron-1639843200-5-etcd-0                                1/1     Running            0          4h25m   10.97.17.194   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-etcd-1                                1/1     Running            0          4h25m   10.97.16.15    qa-node013.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-etcd-2                                1/1     Running            0          4h25m   10.97.16.16    qa-node013.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-milvus-datacoord-78b96cb486-jvrr2     1/1     Running            0          4h25m   10.97.5.48     qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-milvus-datanode-57d888c6f9-t6fft      1/1     Running            0          4h25m   10.97.17.193   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-milvus-indexcoord-88c9f9477-hkrbv     1/1     Running            0          4h25m   10.97.5.46     qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-milvus-indexnode-6898bf744-8r8kk      1/1     Running            0          4h25m   10.97.16.12    qa-node013.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-milvus-proxy-544747c848-lxwzp         1/1     Running            0          4h25m   10.97.5.44     qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-milvus-querycoord-69548458db-bkx84    1/1     Running            0          4h25m   10.97.5.47     qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-milvus-querynode-557c9585d6-qmgzc     1/1     Running            0          4h25m   10.97.20.242   qa-node018.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-milvus-rootcoord-6f948fc778-45kf6     1/1     Running            0          4h25m   10.97.5.45     qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-minio-0                               1/1     Running            0          4h25m   10.97.12.247   qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-minio-1                               1/1     Running            0          4h25m   10.97.12.246   qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-minio-2                               1/1     Running            0          4h25m   10.97.12.244   qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-minio-3                               1/1     Running            0          4h25m   10.97.12.245   qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-autorecovery-5f78cd598dw6srj   1/1     Running            0          4h25m   10.97.9.72     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-bastion-57b6cdd544-5zg4r       1/1     Running            0          4h25m   10.97.7.248    qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-bookkeeper-0                   1/1     Running            0          4h25m   10.97.7.250    qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-bookkeeper-1                   1/1     Running            0          4h23m   10.97.3.185    qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-bookkeeper-2                   1/1     Running            0          4h23m   10.97.9.74     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-broker-5548665df4-mbr4b        1/1     Running            0          4h25m   10.97.3.184    qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-proxy-8b78f6b-qhtxq            2/2     Running            0          4h25m   10.97.8.227    qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-zookeeper-0                    1/1     Running            0          4h25m   10.97.7.249    qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-zookeeper-1                    1/1     Running            0          4h25m   10.97.4.111    qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-zookeeper-2                    1/1     Running            0          4h24m   10.97.9.73     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1639843200-5-pulsar-zookeeper-metadata-ntnnp       0/1     Completed          0          4h25m   10.97.8.228    qa-node006.zilliz.local   <none>           <none>

client pod: benchmark-cron-1639843200-1704316979

client log:

[2021-12-18 17:38:11,015] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639849091.011609904","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 17:32:13.899041', 'RPC start': '2021-12-18 17:32:13.899074', 'RPC error': '2021-12-18 17:38:11.014440'} (pymilvus.client.grpc_handler:84)
[2021-12-18 17:38:11,016] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639849091.011609904","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 17:32:12.407566', 'RPC start': '2021-12-18 17:32:12.407595', 'RPC error': '2021-12-18 17:38:11.016290'} (pymilvus.client.grpc_handler:84)
[2021-12-18 17:45:30,317] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639849530.314556375","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 17:38:14.541425', 'RPC start': '2021-12-18 17:38:14.541457', 'RPC error': '2021-12-18 17:45:30.316537'} (pymilvus.client.grpc_handler:84)
[2021-12-18 17:45:30,318] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639849530.314556375","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 17:38:13.036029', 'RPC start': '2021-12-18 17:38:13.036062', 'RPC error': '2021-12-18 17:45:30.318669'} (pymilvus.client.grpc_handler:84)
[2021-12-18 17:52:50,716] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639849970.712685078","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 17:45:33.870027', 'RPC start': '2021-12-18 17:45:33.870078', 'RPC error': '2021-12-18 17:52:50.715736'} (pymilvus.client.grpc_handler:84)
[2021-12-18 17:52:50,718] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639849970.712685078","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 17:45:32.336395', 'RPC start': '2021-12-18 17:45:32.336415', 'RPC error': '2021-12-18 17:52:50.718112'} (pymilvus.client.grpc_handler:84)
[2021-12-18 17:59:57,467] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639850397.463627472","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 17:52:54.206894', 'RPC start': '2021-12-18 17:52:54.206926', 'RPC error': '2021-12-18 17:59:57.466577'} (pymilvus.client.grpc_handler:84)
[2021-12-18 17:59:57,468] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639850397.463627472","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 17:52:52.736656', 'RPC start': '2021-12-18 17:52:52.736680', 'RPC error': '2021-12-18 17:59:57.468817'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:07:22,001] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639850841.993595181","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:00:01.018225', 'RPC start': '2021-12-18 18:00:01.018268', 'RPC error': '2021-12-18 18:07:22.001187'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:07:22,150] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639850841.993595181","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 17:59:59.487826', 'RPC start': '2021-12-18 17:59:59.487849', 'RPC error': '2021-12-18 18:07:22.150066'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:14:18,614] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639851258.610591272","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:07:25.729464', 'RPC start': '2021-12-18 18:07:25.729500', 'RPC error': '2021-12-18 18:14:18.614165'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:14:18,616] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639851258.610591272","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:07:24.167907', 'RPC start': '2021-12-18 18:07:24.167930', 'RPC error': '2021-12-18 18:14:18.616050'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:21:29,404] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639851689.398588069","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:14:22.041177', 'RPC start': '2021-12-18 18:14:22.041211', 'RPC error': '2021-12-18 18:21:29.403440'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:21:29,405] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639851689.398588069","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:14:20.634801', 'RPC start': '2021-12-18 18:14:20.634831', 'RPC error': '2021-12-18 18:21:29.405607'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:28:54,943] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639852134.937627995","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:21:32.918248', 'RPC start': '2021-12-18 18:21:32.918282', 'RPC error': '2021-12-18 18:28:54.942739'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:28:54,944] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639852134.937627995","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:21:31.424171', 'RPC start': '2021-12-18 18:21:31.424192', 'RPC error': '2021-12-18 18:28:54.944773'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:36:26,536] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639852586.532541059","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:28:58.488997', 'RPC start': '2021-12-18 18:28:58.489030', 'RPC error': '2021-12-18 18:36:26.536120'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:36:26,537] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639852586.532541059","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:28:56.966389', 'RPC start': '2021-12-18 18:28:56.966412', 'RPC error': '2021-12-18 18:36:26.537357'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:43:56,798] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639853036.791571489","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:36:30.103350', 'RPC start': '2021-12-18 18:36:30.103389', 'RPC error': '2021-12-18 18:43:56.797954'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:43:56,800] [   ERROR] - 
Addr [benchmark-cron-1639843200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639853036.791571489","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-18 18:36:28.557691', 'RPC start': '2021-12-18 18:36:28.557712', 'RPC error': '2021-12-18 18:43:56.799965'} (pymilvus.client.grpc_handler:84)
[2021-12-18 18:43:56,810] [   ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1048, in search
    partition_names, output_fields, timeout, round_decimal, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 643, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 605, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 593, in _execute_search_requests
    response = ft.result()
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639853036.791571489","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 94, in run_suite
    result = runner.run_case(case_metric, **case)
  File "/src/milvus_benchmark/runners/accuracy.py", line 270, in run_case
    guarantee_timestamp=case_param["guarantee_timestamp"])
  File "/src/milvus_benchmark/client.py", line 49, in wrapper
    result = func(*args, **kwargs)
  File "/src/milvus_benchmark/client.py", line 322, in query
    result = self._milvus.search(tmp_collection_name, **params)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 53, in handler
    raise BaseException(1, "rpc timeout")
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>
 (milvus_benchmark.main:97)
[2021-12-18 18:43:56,811] [    INFO] - None (milvus_benchmark.main:98)
[2021-12-18 18:43:56,812] [   DEBUG] - {'type': 'ann_accuracy', 'value': {}} (milvus_benchmark.main:106)
[2021-12-18 18:43:56,812] [   DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1639843589, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7f56c4498eb8>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7f56c4498dd8>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7f56c4498cf8>, 'status': 'RUN_FAILED', 'err_message': '<BaseException: (code=1, message=rpc timeout)>\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1048, in search\n    partition_names, output_fields, timeout, round_decimal, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler\n    raise e\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 643, in search\n    return self._execute_search_requests(requests, timeout, **_kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler\n    raise e\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 605, in _execute_search_requests\n    raise pre_err\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 593, in _execute_search_requests\n    response = ft.result()\n  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result\n    raise self\ngrpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:\n\tstatus = StatusCode.DEADLINE_EXCEEDED\n\tdetails = "Deadline Exceeded"\n\tdebug_error_string = "{"created":"@1639853036.791571489","description":"Error received from peer ipv4:10.96.226.67:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"\n>\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "main.py", line 94, in run_suite\n    result = runner.run_case(case_metric, **case)\n  File "/src/milvus_benchmark/runners/accuracy.py", line 270, in run_case\n    guarantee_timestamp=case_param["guarantee_timestamp"])\n  File "/src/milvus_benchmark/client.py", line 49, in wrapper\n    result = func(*args, **kwargs)\n  File "/src/milvus_benchmark/client.py", line 322, in query\n    result = self._milvus.search(tmp_collection_name, **params)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 53, in handler\n    raise BaseException(1, "rpc timeout")\npymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>\n', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {'index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 512}, 'filter': [], 'guarantee_timestamp': None}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {}}, 'datetime': '2021-12-18 16:06:29.681077', 'type': 'metric'} (milvus_benchmark.metric.api:29)
[2021-12-18 18:43:56,836] [   DEBUG] - {'_version': '0.1', '_type': 'metric', 'run_id': 1639843589, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7f56c555ac50>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7f56c555af98>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7f56c555acc0>, 'status': 'RUN_FAILED', 'err_message': '', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {}, 'search': None, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {}}, 'datetime': '2021-12-18 16:06:29.681077', 'type': 'metric'} (milvus_benchmark.metric.api:29)

wangting0128 commented 2 years ago

argo task: benchmark-cron-1639929600

test yaml: client-configmap:client-acc-glove-ivf-flat server-configmap:server-cluster-8c16m

server:

NAME                                                              READY   STATUS      RESTARTS   AGE     IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-cron-1639929600-5-etcd-0                                1/1     Running     0          4h21m   10.97.17.244   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-etcd-1                                1/1     Running     0          4h21m   10.97.16.40    qa-node013.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-etcd-2                                1/1     Running     0          4h21m   10.97.17.245   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-milvus-datacoord-64446cc6c-g257d      1/1     Running     0          4h21m   10.97.5.110    qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-milvus-datanode-6fcb998d68-9zmfq      1/1     Running     0          4h21m   10.97.19.130   qa-node016.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-milvus-indexcoord-5fd96d6b98-2k6x9    1/1     Running     0          4h21m   10.97.5.111    qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-milvus-indexnode-5fff847f49-n89pp     1/1     Running     0          4h21m   10.97.13.40    qa-node010.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-milvus-proxy-78455f9955-2pw2q         1/1     Running     0          4h21m   10.97.5.113    qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-milvus-querycoord-8774f7bb5-wv29z     1/1     Running     0          4h21m   10.97.5.112    qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-milvus-querynode-679c99d5cf-dk2mw     1/1     Running     0          4h21m   10.97.15.126   qa-node012.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-milvus-rootcoord-fff7c64c-4cn77       1/1     Running     0          4h21m   10.97.8.231    qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-minio-0                               1/1     Running     0          4h21m   10.97.19.125   qa-node016.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-minio-1                               1/1     Running     0          4h21m   10.97.19.128   qa-node016.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-minio-2                               1/1     Running     0          4h21m   10.97.19.127   qa-node016.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-minio-3                               1/1     Running     0          4h21m   10.97.19.126   qa-node016.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-autorecovery-69bc499d-8c64g    1/1     Running     0          4h21m   10.97.9.80     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-bastion-766db568fb-nlv5v       1/1     Running     0          4h21m   10.97.7.3      qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-bookkeeper-0                   1/1     Running     0          4h21m   10.97.7.6      qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-bookkeeper-1                   1/1     Running     0          4h20m   10.97.8.234    qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-bookkeeper-2                   1/1     Running     0          4h19m   10.97.7.7      qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-broker-6768fd4857-9wsgv        1/1     Running     0          4h21m   10.97.15.125   qa-node012.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-proxy-785ddcff87-7qwdm         2/2     Running     0          4h21m   10.97.4.119    qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-zookeeper-0                    1/1     Running     0          4h21m   10.97.7.5      qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-zookeeper-1                    1/1     Running     0          4h21m   10.97.8.233    qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-zookeeper-2                    1/1     Running     0          4h21m   10.97.9.82     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1639929600-5-pulsar-zookeeper-metadata-8k5xk       0/1     Completed   0          4h21m   10.97.7.4      qa-node005.zilliz.local   <none>           <none>

client pod: benchmark-cron-1639929600-794679664

client log:

[2021-12-19 17:45:33,059] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639935933.048108023","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 17:39:31.503223', 'RPC start': '2021-12-19 17:39:31.503247', 'RPC error': '2021-12-19 17:45:33.058411'} (pymilvus.client.grpc_handler:84)
[2021-12-19 17:45:33,061] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639935933.048108023","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 17:39:30.086354', 'RPC start': '2021-12-19 17:39:30.086377', 'RPC error': '2021-12-19 17:45:33.061051'} (pymilvus.client.grpc_handler:84)
[2021-12-19 17:53:18,010] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639936398.002114591","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 17:45:36.551359', 'RPC start': '2021-12-19 17:45:36.551385', 'RPC error': '2021-12-19 17:53:18.009987'} (pymilvus.client.grpc_handler:84)
[2021-12-19 17:53:18,011] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639936398.002114591","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 17:45:35.084311', 'RPC start': '2021-12-19 17:45:35.084344', 'RPC error': '2021-12-19 17:53:18.011707'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:01:03,830] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639936863.825148686","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 17:53:21.549965', 'RPC start': '2021-12-19 17:53:21.549998', 'RPC error': '2021-12-19 18:01:03.829708'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:01:03,831] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639936863.825148686","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 17:53:20.036718', 'RPC start': '2021-12-19 17:53:20.036751', 'RPC error': '2021-12-19 18:01:03.831189'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:07:10,681] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639937230.668159584","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:01:07.432177', 'RPC start': '2021-12-19 18:01:07.432264', 'RPC error': '2021-12-19 18:07:10.679770'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:07:10,684] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639937230.668159584","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:01:05.849944', 'RPC start': '2021-12-19 18:01:05.849966', 'RPC error': '2021-12-19 18:07:10.684658'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:14:54,566] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639937694.560133300","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:07:14.228783', 'RPC start': '2021-12-19 18:07:14.228811', 'RPC error': '2021-12-19 18:14:54.566162'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:14:54,568] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639937694.560133300","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:07:12.706388', 'RPC start': '2021-12-19 18:07:12.706410', 'RPC error': '2021-12-19 18:14:54.568176'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:22:42,651] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639938162.643163301","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:14:58.057888', 'RPC start': '2021-12-19 18:14:58.057938', 'RPC error': '2021-12-19 18:22:42.650910'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:22:42,653] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639938162.643163301","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:14:56.591100', 'RPC start': '2021-12-19 18:14:56.591124', 'RPC error': '2021-12-19 18:22:42.653112'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:30:31,721] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639938631.708262829","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:22:46.251090', 'RPC start': '2021-12-19 18:22:46.251185', 'RPC error': '2021-12-19 18:30:31.720918'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:30:31,723] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639938631.708262829","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:22:44.700184', 'RPC start': '2021-12-19 18:22:44.700211', 'RPC error': '2021-12-19 18:30:31.723620'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:38:20,001] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639939099.996127455","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:30:35.257338', 'RPC start': '2021-12-19 18:30:35.257373', 'RPC error': '2021-12-19 18:38:20.000322'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:38:20,003] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639939099.996127455","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:30:33.746358', 'RPC start': '2021-12-19 18:30:33.746381', 'RPC error': '2021-12-19 18:38:20.003822'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:46:09,409] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639939569.399104631","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:38:23.519547', 'RPC start': '2021-12-19 18:38:23.519580', 'RPC error': '2021-12-19 18:46:09.408959'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:46:09,411] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639939569.399104631","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:38:22.026333', 'RPC start': '2021-12-19 18:38:22.026356', 'RPC error': '2021-12-19 18:46:09.411392'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:53:53,825] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639940033.815157263","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:46:12.934487', 'RPC start': '2021-12-19 18:46:12.934576', 'RPC error': '2021-12-19 18:53:53.824777'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:53:53,826] [   ERROR] - 
Addr [benchmark-cron-1639929600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639940033.815157263","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-19 18:46:11.429362', 'RPC start': '2021-12-19 18:46:11.429382', 'RPC error': '2021-12-19 18:53:53.826815'} (pymilvus.client.grpc_handler:84)
[2021-12-19 18:53:53,834] [   ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1048, in search
    partition_names, output_fields, timeout, round_decimal, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 643, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 605, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 593, in _execute_search_requests
    response = ft.result()
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1639940033.815157263","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 94, in run_suite
    result = runner.run_case(case_metric, **case)
  File "/src/milvus_benchmark/runners/accuracy.py", line 270, in run_case
    guarantee_timestamp=case_param["guarantee_timestamp"])
  File "/src/milvus_benchmark/client.py", line 49, in wrapper
    result = func(*args, **kwargs)
  File "/src/milvus_benchmark/client.py", line 322, in query
    result = self._milvus.search(tmp_collection_name, **params)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 53, in handler
    raise BaseException(1, "rpc timeout")
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>
 (milvus_benchmark.main:97)
[2021-12-19 18:53:53,835] [    INFO] - None (milvus_benchmark.main:98)
[2021-12-19 18:53:53,836] [   DEBUG] - {'type': 'ann_accuracy', 'value': {}} (milvus_benchmark.main:106)
[2021-12-19 18:53:53,837] [   DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1639929939, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7fc160369ef0>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7fc160369e10>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7fc160369d30>, 'status': 'RUN_FAILED', 'err_message': '<BaseException: (code=1, message=rpc timeout)>\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1048, in search\n    partition_names, output_fields, timeout, round_decimal, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler\n    raise e\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 643, in search\n    return self._execute_search_requests(requests, timeout, **_kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler\n    raise e\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 605, in _execute_search_requests\n    raise pre_err\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 593, in _execute_search_requests\n    response = ft.result()\n  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result\n    raise self\ngrpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:\n\tstatus = StatusCode.DEADLINE_EXCEEDED\n\tdetails = "Deadline Exceeded"\n\tdebug_error_string = "{"created":"@1639940033.815157263","description":"Error received from peer ipv4:10.96.112.45:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"\n>\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "main.py", line 94, in run_suite\n    result = runner.run_case(case_metric, **case)\n  File "/src/milvus_benchmark/runners/accuracy.py", line 270, in run_case\n    guarantee_timestamp=case_param["guarantee_timestamp"])\n  File "/src/milvus_benchmark/client.py", line 49, in wrapper\n    result = func(*args, **kwargs)\n  File "/src/milvus_benchmark/client.py", line 322, in query\n    result = self._milvus.search(tmp_collection_name, **params)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 53, in handler\n    raise BaseException(1, "rpc timeout")\npymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>\n', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {'index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 512}, 'filter': [], 'guarantee_timestamp': None}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {}}, 'datetime': '2021-12-19 16:05:39.160720', 'type': 'metric'} (milvus_benchmark.metric.api:29)
[2021-12-19 18:53:53,849] [   DEBUG] - {'_version': '0.1', '_type': 'metric', 'run_id': 1639929939, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7fc16242dc50>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7fc16242df98>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7fc16242dcc0>, 'status': 'RUN_FAILED', 'err_message': '', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {}, 'search': None, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {}}, 'datetime': '2021-12-19 16:05:39.160720', 'type': 'metric'} (milvus_benchmark.metric.api:29)

xiaocai2333 commented 2 years ago

There is two problems: about @jingkl pasted, it is because querynode paniced, it seem to be related to 'cgocall', but I can't reproduce it.

and about @wangting0128 pasted, it is because the search time is longer than normal.

yanliang567 commented 2 years ago

remove from urgent as it an occasional issue, and not reproduce recently.

wangting0128 commented 2 years ago

argo task: benchmark-cron-1640275200

test yaml: client-configmap:client-acc-glove-ivf-flat server-configmap:server-cluster-8c16m

server:

NAME                                                              READY   STATUS      RESTARTS   AGE     IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-cron-1640275200-5-etcd-0                                1/1     Running     0          3h57m   10.97.17.191   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-etcd-1                                1/1     Running     0          3h57m   10.97.16.203   qa-node013.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-etcd-2                                1/1     Running     0          3h57m   10.97.17.193   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-milvus-datacoord-75ddf8b9c9-84l6s     1/1     Running     0          3h57m   10.97.5.245    qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-milvus-datanode-7868b844fd-fkhrk      1/1     Running     0          3h57m   10.97.13.98    qa-node010.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-milvus-indexcoord-59d6849d8-45x77     1/1     Running     0          3h57m   10.97.5.247    qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-milvus-indexnode-868876894f-fn994     1/1     Running     0          3h57m   10.97.12.99    qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-milvus-proxy-86966d9dcf-szv44         1/1     Running     0          3h57m   10.97.5.246    qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-milvus-querycoord-7d494499c9-d76qt    1/1     Running     0          3h57m   10.97.3.240    qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-milvus-querynode-98448bc5-fcvbp       1/1     Running     0          3h57m   10.97.15.223   qa-node012.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-milvus-rootcoord-6868fc75c9-fnnb9     1/1     Running     0          3h57m   10.97.9.240    qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-minio-0                               1/1     Running     0          3h57m   10.97.12.97    qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-minio-1                               1/1     Running     0          3h57m   10.97.12.95    qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-minio-2                               1/1     Running     0          3h57m   10.97.12.96    qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-minio-3                               1/1     Running     0          3h57m   10.97.12.98    qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-autorecovery-569fdf8fbb4lgkd   1/1     Running     0          3h57m   10.97.3.242    qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-bastion-64bcbf4fb4-jbkzk       1/1     Running     0          3h57m   10.97.3.241    qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-bookkeeper-0                   1/1     Running     0          3h57m   10.97.7.92     qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-bookkeeper-1                   1/1     Running     0          3h55m   10.97.7.93     qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-bookkeeper-2                   1/1     Running     0          3h54m   10.97.3.244    qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-broker-bb498c49c-5c7x6         1/1     Running     0          3h57m   10.97.4.207    qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-proxy-66fc5c69dc-jwlcc         2/2     Running     0          3h57m   10.97.9.241    qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-zookeeper-0                    1/1     Running     0          3h57m   10.97.3.243    qa-node001.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-zookeeper-1                    1/1     Running     0          3h56m   10.97.8.62     qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-zookeeper-2                    1/1     Running     0          3h56m   10.97.9.242    qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1640275200-5-pulsar-zookeeper-metadata-h5t2g       0/1     Completed   0          3h57m   10.97.4.206    qa-node002.zilliz.local   <none>           <none>

client pod: benchmark-cron-1640275200-3688309676

client log:

[2021-12-23 17:50:28,885] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640281828.882017277","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 17:44:29.990226', 'RPC start': '2021-12-23 17:44:29.990271', 'RPC error': '2021-12-23 17:50:28.885317'} (pymilvus.client.grpc_handler:84)
[2021-12-23 17:50:28,886] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640281828.882017277","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 17:44:28.714664', 'RPC start': '2021-12-23 17:44:28.714676', 'RPC error': '2021-12-23 17:50:28.886664'} (pymilvus.client.grpc_handler:84)
[2021-12-23 17:58:12,668] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640282292.666395829","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 17:50:32.322455', 'RPC start': '2021-12-23 17:50:32.322484', 'RPC error': '2021-12-23 17:58:12.668314'} (pymilvus.client.grpc_handler:84)
[2021-12-23 17:58:12,670] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640282292.666395829","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 17:50:30.908663', 'RPC start': '2021-12-23 17:50:30.908684', 'RPC error': '2021-12-23 17:58:12.670323'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:05:52,040] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640282752.034060794","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 17:58:16.322194', 'RPC start': '2021-12-23 17:58:16.322221', 'RPC error': '2021-12-23 18:05:52.039603'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:05:52,042] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640282752.034060794","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 17:58:14.690397', 'RPC start': '2021-12-23 17:58:14.690420', 'RPC error': '2021-12-23 18:05:52.042540'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:13:31,200] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640283211.196949332","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:05:55.712575', 'RPC start': '2021-12-23 18:05:55.712602', 'RPC error': '2021-12-23 18:13:31.199608'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:13:31,201] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640283211.196949332","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:05:54.061678', 'RPC start': '2021-12-23 18:05:54.061707', 'RPC error': '2021-12-23 18:13:31.201298'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:21:09,480] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640283669.471108001","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:13:34.703248', 'RPC start': '2021-12-23 18:13:34.703277', 'RPC error': '2021-12-23 18:21:09.479807'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:21:09,482] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640283669.471108001","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:13:33.219596', 'RPC start': '2021-12-23 18:13:33.219617', 'RPC error': '2021-12-23 18:21:09.482707'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:28:51,856] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640284131.851540263","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:21:13.060623', 'RPC start': '2021-12-23 18:21:13.060664', 'RPC error': '2021-12-23 18:28:51.855507'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:28:51,866] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640284131.851540263","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:21:11.501811', 'RPC start': '2021-12-23 18:21:11.501836', 'RPC error': '2021-12-23 18:28:51.866635'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:36:28,434] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640284588.430047650","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:28:55.267013', 'RPC start': '2021-12-23 18:28:55.267038', 'RPC error': '2021-12-23 18:36:28.433725'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:36:28,435] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640284588.430047650","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:28:53.887839', 'RPC start': '2021-12-23 18:28:53.887868', 'RPC error': '2021-12-23 18:36:28.435479'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:44:06,993] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640285046.982907261","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:36:31.824750', 'RPC start': '2021-12-23 18:36:31.824790', 'RPC error': '2021-12-23 18:44:06.988730'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:44:06,994] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640285046.982907261","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:36:30.454441', 'RPC start': '2021-12-23 18:36:30.454459', 'RPC error': '2021-12-23 18:44:06.994664'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:51:42,779] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640285502.771046622","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:44:10.705775', 'RPC start': '2021-12-23 18:44:10.705802', 'RPC error': '2021-12-23 18:51:42.779390'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:51:42,780] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640285502.771046622","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:44:09.012160', 'RPC start': '2021-12-23 18:44:09.012179', 'RPC error': '2021-12-23 18:51:42.780835'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:59:21,790] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640285961.774921538","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:51:46.173831', 'RPC start': '2021-12-23 18:51:46.173859', 'RPC error': '2021-12-23 18:59:21.790278'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:59:21,791] [   ERROR] - 
Addr [benchmark-cron-1640275200-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640285961.774921538","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-23 18:51:44.801774', 'RPC start': '2021-12-23 18:51:44.801791', 'RPC error': '2021-12-23 18:59:21.791484'} (pymilvus.client.grpc_handler:84)
[2021-12-23 18:59:21,799] [   ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1048, in search
    partition_names, output_fields, timeout, round_decimal, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 643, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 605, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 593, in _execute_search_requests
    response = ft.result()
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640285961.774921538","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 94, in run_suite
    result = runner.run_case(case_metric, **case)
  File "/src/milvus_benchmark/runners/accuracy.py", line 270, in run_case
    guarantee_timestamp=case_param["guarantee_timestamp"])
  File "/src/milvus_benchmark/client.py", line 49, in wrapper
    result = func(*args, **kwargs)
  File "/src/milvus_benchmark/client.py", line 322, in query
    result = self._milvus.search(tmp_collection_name, **params)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 53, in handler
    raise BaseException(1, "rpc timeout")
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>
 (milvus_benchmark.main:97)
[2021-12-23 18:59:21,799] [    INFO] - None (milvus_benchmark.main:98)
[2021-12-23 18:59:21,800] [   DEBUG] - {'type': 'ann_accuracy', 'value': {}} (milvus_benchmark.main:106)
[2021-12-23 18:59:21,801] [   DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1640275870, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7fde2c979278>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7fde2c9791d0>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7fde2c979240>, 'status': 'RUN_FAILED', 'err_message': '<BaseException: (code=1, message=rpc timeout)>\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1048, in search\n    partition_names, output_fields, timeout, round_decimal, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler\n    raise e\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 643, in search\n    return self._execute_search_requests(requests, timeout, **_kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler\n    raise e\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 605, in _execute_search_requests\n    raise pre_err\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 593, in _execute_search_requests\n    response = ft.result()\n  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result\n    raise self\ngrpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:\n\tstatus = StatusCode.DEADLINE_EXCEEDED\n\tdetails = "Deadline Exceeded"\n\tdebug_error_string = "{"created":"@1640285961.774921538","description":"Error received from peer ipv4:10.96.217.119:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"\n>\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "main.py", line 94, in run_suite\n    result = runner.run_case(case_metric, **case)\n  File "/src/milvus_benchmark/runners/accuracy.py", line 270, in run_case\n    guarantee_timestamp=case_param["guarantee_timestamp"])\n  File "/src/milvus_benchmark/client.py", line 49, in wrapper\n    result = func(*args, **kwargs)\n  File "/src/milvus_benchmark/client.py", line 322, in query\n    result = self._milvus.search(tmp_collection_name, **params)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 53, in handler\n    raise BaseException(1, "rpc timeout")\npymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>\n', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {'index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 512}, 'filter': [], 'guarantee_timestamp': None}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {}}, 'datetime': '2021-12-23 16:11:10.306306', 'type': 'metric'} (milvus_benchmark.metric.api:29)
[2021-12-23 18:59:21,816] [   DEBUG] - {'_version': '0.1', '_type': 'metric', 'run_id': 1640275870, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7fde2e9c8978>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7fde2e9c8550>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7fde2e9c8b70>, 'status': 'RUN_FAILED', 'err_message': '', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {}, 'search': None, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {}}, 'datetime': '2021-12-23 16:11:10.306306', 'type': 'metric'} (milvus_benchmark.metric.api:29)

xiaocai2333 commented 2 years ago

It should be the cpu configuration of the machine where the querynode is located. The querynode is allocated to qa-node12 every time an error occurs. And I also did a test, and it is indeed slower to do queries on this machine than on other machines. @LoveEachDay Please check it.

When top=10, nq=1000, sdk will divide the big request into 8 small requests whose nq = 1310 (one of the requests nq = 830), and proxy will receive 8 requests at the same time. On qa-node12, one request will cost about 120 seconds, and these requests are executed serially on querynode. So the last request will time out(request timeout = 300).

wangting0128 commented 2 years ago

argo task: benchmark-cron-1640361600

test yaml: client-configmap:client-acc-glove-ivf-flat server-configmap:server-cluster-8c16m

server:

NAME                                                              READY   STATUS      RESTARTS   AGE    IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-cron-1640361600-5-etcd-0                                1/1     Running     0          4h3m   10.97.17.241   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-etcd-1                                1/1     Running     0          4h3m   10.97.16.102   qa-node013.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-etcd-2                                1/1     Running     0          4h3m   10.97.17.243   qa-node014.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-milvus-datacoord-5bfb58748f-89w29     1/1     Running     0          4h3m   10.97.5.88     qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-milvus-datanode-855bdb5bc4-c2gdh      1/1     Running     0          4h3m   10.97.11.71    qa-node009.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-milvus-indexcoord-678c54f544-4w4jg    1/1     Running     0          4h3m   10.97.5.86     qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-milvus-indexnode-68c7777fcf-x8bs7     1/1     Running     0          4h3m   10.97.14.189   qa-node011.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-milvus-proxy-5c77f859d9-ncm8n         1/1     Running     0          4h3m   10.97.7.111    qa-node005.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-milvus-querycoord-84684df6c9-xcpfp    1/1     Running     0          4h3m   10.97.5.87     qa-node003.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-milvus-querynode-8655cdc4c7-lz7sv     1/1     Running     0          4h3m   10.97.20.252   qa-node018.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-milvus-rootcoord-6ffdbbc497-mpnq9     1/1     Running     0          4h3m   10.97.9.51     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-minio-0                               1/1     Running     0          4h3m   10.97.20.250   qa-node018.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-minio-1                               1/1     Running     0          4h3m   10.97.20.251   qa-node018.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-minio-2                               1/1     Running     0          4h3m   10.97.19.253   qa-node016.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-minio-3                               1/1     Running     0          4h3m   10.97.12.152   qa-node015.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-autorecovery-5c657cd46-snbzr   1/1     Running     0          4h3m   10.97.9.53     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-bastion-c668fbb4d-m5qqn        1/1     Running     0          4h3m   10.97.9.54     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-bookkeeper-0                   1/1     Running     0          4h3m   10.97.10.158   qa-node008.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-bookkeeper-1                   1/1     Running     0          4h1m   10.97.13.117   qa-node010.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-bookkeeper-2                   1/1     Running     0          4h1m   10.97.8.96     qa-node006.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-broker-67d47c9bc5-j64vz        1/1     Running     0          4h3m   10.97.13.115   qa-node010.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-proxy-b4776ddb5-8df8b          2/2     Running     0          4h3m   10.97.10.157   qa-node008.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-zookeeper-0                    1/1     Running     0          4h3m   10.97.4.17     qa-node002.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-zookeeper-1                    1/1     Running     0          4h2m   10.97.9.55     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-zookeeper-2                    1/1     Running     0          4h2m   10.97.9.57     qa-node007.zilliz.local   <none>           <none>
benchmark-cron-1640361600-5-pulsar-zookeeper-metadata-4zxlq       0/1     Completed   0          4h3m   10.97.9.52     qa-node007.zilliz.local   <none>           <none>

client pod: benchmark-cron-1640361600-3346309166

client logs:

[2021-12-24 17:43:22,043] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640367802.042378899","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 17:37:32.425726', 'RPC start': '2021-12-24 17:37:32.425752', 'RPC error': '2021-12-24 17:43:22.043229'} (pymilvus.client.grpc_handler:84)
[2021-12-24 17:43:22,044] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640367802.042378899","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 17:37:30.952374', 'RPC start': '2021-12-24 17:37:30.952390', 'RPC error': '2021-12-24 17:43:22.044675'} (pymilvus.client.grpc_handler:84)
[2021-12-24 17:49:37,284] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640368177.283368948","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 17:43:25.360880', 'RPC start': '2021-12-24 17:43:25.360901', 'RPC error': '2021-12-24 17:49:37.284447'} (pymilvus.client.grpc_handler:84)
[2021-12-24 17:49:37,286] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640368177.283368948","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 17:43:24.062284', 'RPC start': '2021-12-24 17:43:24.062301', 'RPC error': '2021-12-24 17:49:37.286030'} (pymilvus.client.grpc_handler:84)
[2021-12-24 17:55:54,255] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640368554.253378453","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 17:49:40.757944', 'RPC start': '2021-12-24 17:49:40.757965', 'RPC error': '2021-12-24 17:55:54.254670'} (pymilvus.client.grpc_handler:84)
[2021-12-24 17:55:54,256] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640368554.253378453","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 17:49:39.304200', 'RPC start': '2021-12-24 17:49:39.304214', 'RPC error': '2021-12-24 17:55:54.256757'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:02:10,729] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640368930.728374148","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 17:55:57.810852', 'RPC start': '2021-12-24 17:55:57.810875', 'RPC error': '2021-12-24 18:02:10.729150'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:02:10,730] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640368930.728374148","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 17:55:56.296709', 'RPC start': '2021-12-24 17:55:56.296730', 'RPC error': '2021-12-24 18:02:10.730200'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:08:26,104] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640369306.102333799","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:02:14.342736', 'RPC start': '2021-12-24 18:02:14.342757', 'RPC error': '2021-12-24 18:08:26.103976'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:08:26,107] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640369306.102333799","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:02:12.749237', 'RPC start': '2021-12-24 18:02:12.749263', 'RPC error': '2021-12-24 18:08:26.107901'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:14:41,414] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640369681.413375735","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:08:29.612452', 'RPC start': '2021-12-24 18:08:29.612473', 'RPC error': '2021-12-24 18:14:41.414472'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:14:41,416] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640369681.413375735","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:08:28.124656', 'RPC start': '2021-12-24 18:08:28.124674', 'RPC error': '2021-12-24 18:14:41.416337'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:20:55,510] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640370055.509377258","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:14:44.887395', 'RPC start': '2021-12-24 18:14:44.887415', 'RPC error': '2021-12-24 18:20:55.510234'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:20:55,511] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640370055.509377258","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:14:43.437511', 'RPC start': '2021-12-24 18:14:43.437527', 'RPC error': '2021-12-24 18:20:55.511417'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:27:12,865] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640370432.863325633","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:20:58.965908', 'RPC start': '2021-12-24 18:20:58.965926', 'RPC error': '2021-12-24 18:27:12.864699'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:27:12,866] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640370432.863325633","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:20:57.530834', 'RPC start': '2021-12-24 18:20:57.530851', 'RPC error': '2021-12-24 18:27:12.866841'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:33:29,320] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640370809.318376746","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:27:16.486909', 'RPC start': '2021-12-24 18:27:16.486938', 'RPC error': '2021-12-24 18:33:29.319663'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:33:29,321] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640370809.318376746","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:27:14.883416', 'RPC start': '2021-12-24 18:27:14.883433', 'RPC error': '2021-12-24 18:33:29.321485'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:39:46,158] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] _execute_search_requests
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640371186.157368288","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:33:32.716946', 'RPC start': '2021-12-24 18:33:32.716966', 'RPC error': '2021-12-24 18:39:46.158472'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:39:46,161] [   ERROR] - 
Addr [benchmark-cron-1640361600-5-milvus.qa-milvus.svc.cluster.local:19530] search
RPC error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640371186.157368288","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
    {'API start': '2021-12-24 18:33:31.352687', 'RPC start': '2021-12-24 18:33:31.352707', 'RPC error': '2021-12-24 18:39:46.161313'} (pymilvus.client.grpc_handler:84)
[2021-12-24 18:39:46,168] [   ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1048, in search
    partition_names, output_fields, timeout, round_decimal, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 643, in search
    return self._execute_search_requests(requests, timeout, **_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler
    raise e
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 605, in _execute_search_requests
    raise pre_err
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 593, in _execute_search_requests
    response = ft.result()
  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1640371186.157368288","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 94, in run_suite
    result = runner.run_case(case_metric, **case)
  File "/src/milvus_benchmark/runners/accuracy.py", line 270, in run_case
    guarantee_timestamp=case_param["guarantee_timestamp"])
  File "/src/milvus_benchmark/client.py", line 49, in wrapper
    result = func(*args, **kwargs)
  File "/src/milvus_benchmark/client.py", line 322, in query
    result = self._milvus.search(tmp_collection_name, **params)
  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 53, in handler
    raise BaseException(1, "rpc timeout")
pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>
 (milvus_benchmark.main:97)
[2021-12-24 18:39:46,191] [    INFO] - None (milvus_benchmark.main:98)
[2021-12-24 18:39:46,191] [   DEBUG] - {'type': 'ann_accuracy', 'value': {}} (milvus_benchmark.main:106)
[2021-12-24 18:39:46,192] [   DEBUG] - {'_version': '0.1', '_type': 'case', 'run_id': 1640362256, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7f24058671d0>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7f2405867390>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7f2405867278>, 'status': 'RUN_FAILED', 'err_message': '<BaseException: (code=1, message=rpc timeout)>\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 42, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 1048, in search\n    partition_names, output_fields, timeout, round_decimal, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler\n    raise e\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 104, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 643, in search\n    return self._execute_search_requests(requests, timeout, **_kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 85, in handler\n    raise e\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 67, in handler\n    return func(self, *args, **kwargs)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 605, in _execute_search_requests\n    raise pre_err\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/grpc_handler.py", line 593, in _execute_search_requests\n    response = ft.result()\n  File "/usr/local/lib/python3.6/site-packages/grpc/_channel.py", line 744, in result\n    raise self\ngrpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:\n\tstatus = StatusCode.DEADLINE_EXCEEDED\n\tdetails = "Deadline Exceeded"\n\tdebug_error_string = "{"created":"@1640371186.157368288","description":"Error received from peer ipv4:10.96.235.145:19530","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}"\n>\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "main.py", line 94, in run_suite\n    result = runner.run_case(case_metric, **case)\n  File "/src/milvus_benchmark/runners/accuracy.py", line 270, in run_case\n    guarantee_timestamp=case_param["guarantee_timestamp"])\n  File "/src/milvus_benchmark/client.py", line 49, in wrapper\n    result = func(*args, **kwargs)\n  File "/src/milvus_benchmark/client.py", line 322, in query\n    result = self._milvus.search(tmp_collection_name, **params)\n  File "/usr/local/lib/python3.6/site-packages/pymilvus/client/stub.py", line 53, in handler\n    raise BaseException(1, "rpc timeout")\npymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=rpc timeout)>\n', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {'index_type': 'ivf_flat', 'index_param': {'nlist': 1024}}, 'search': {'nq': 10000, 'topk': 10, 'search_param': {'nprobe': 512}, 'filter': [], 'guarantee_timestamp': None}, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {}}, 'datetime': '2021-12-24 16:10:56.793857', 'type': 'metric'} (milvus_benchmark.metric.api:29)
[2021-12-24 18:39:46,215] [   DEBUG] - {'_version': '0.1', '_type': 'metric', 'run_id': 1640362256, 'mode': 'local', 'server': <milvus_benchmark.metrics.models.server.Server object at 0x7f24078b9908>, 'hardware': <milvus_benchmark.metrics.models.hardware.Hardware object at 0x7f24078b9da0>, 'env': <milvus_benchmark.metrics.models.env.Env object at 0x7f24078b9ba8>, 'status': 'RUN_FAILED', 'err_message': '', 'collection': {'dimension': 200, 'metric_type': 'ip', 'dataset_name': 'glove_200_angular', 'shards_num': None}, 'index': {}, 'search': None, 'run_params': None, 'metrics': {'type': 'ann_accuracy', 'value': {}}, 'datetime': '2021-12-24 16:10:56.793857', 'type': 'metric'} (milvus_benchmark.metric.api:29)

xiaocai2333 commented 2 years ago

Can this problem be reproduced? @wangting0128

wangting0128 commented 2 years ago

argo task: benchmark-tag-z4nwl

test yaml: client-configmap:client-random-locust-search-84h-1b server-configmap:server-cluster-8c64m-datanode2-indexnode4-querynode6

server:

NAME                                                              READY   STATUS      RESTARTS   AGE    IP             NODE                      NOMINATED NODE   READINESS GATES
benchmark-tag-z4nwl-1-etcd-0                                      1/1     Running     0          46h    10.97.17.83    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-etcd-1                                      1/1     Running     0          46h    10.97.16.30    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-etcd-2                                      1/1     Running     0          46h    10.97.16.26    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-datacoord-94479dd55-pkvtw            1/1     Running     0          46h    10.97.17.80    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-datanode-7b685d54fb-mqbg5            1/1     Running     0          46h    10.97.17.78    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-datanode-7b685d54fb-rbxh5            1/1     Running     0          46h    10.97.17.77    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-indexcoord-79dbd48dbd-tvq47          1/1     Running     0          46h    10.97.12.122   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-indexnode-fb6fc64c-628p5             1/1     Running     0          46h    10.97.19.56    qa-node016.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-indexnode-fb6fc64c-7s4wt             1/1     Running     0          46h    10.97.16.22    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-indexnode-fb6fc64c-7wc9s             1/1     Running     0          46h    10.97.12.123   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-indexnode-fb6fc64c-blrnm             1/1     Running     0          46h    10.97.17.79    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-proxy-5576964556-mkhjj               1/1     Running     0          46h    10.97.17.75    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-querycoord-b477bf6bc-qlzhv           1/1     Running     0          46h    10.97.17.76    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-querynode-b9cd75857-6z78l            1/1     Running     0          46h    10.97.19.58    qa-node016.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-querynode-b9cd75857-7nmbr            1/1     Running     0          46h    10.97.12.124   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-querynode-b9cd75857-fgvc9            1/1     Running     0          46h    10.97.16.23    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-querynode-b9cd75857-slzwb            1/1     Running     0          46h    10.97.16.24    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-querynode-b9cd75857-t9kgm            1/1     Running     0          46h    10.97.17.82    qa-node014.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-querynode-b9cd75857-wg9d4            1/1     Running     1          46h    10.97.20.8     qa-node018.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-milvus-rootcoord-6bb7cff967-dmtzv           1/1     Running     0          46h    10.97.19.55    qa-node016.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-minio-0                                     1/1     Running     0          46h    10.97.19.60    qa-node016.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-minio-1                                     1/1     Running     0          46h    10.97.16.29    qa-node013.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-minio-2                                     1/1     Running     0          46h    10.97.19.61    qa-node016.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-minio-3                                     1/1     Running     0          46h    10.97.12.126   qa-node015.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-autorecovery-59d6cd8966-5hw9s        1/1     Running     0          46h    10.97.20.3     qa-node018.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-bastion-86656b58d4-dpf8q             1/1     Running     0          46h    10.97.20.7     qa-node018.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-bookkeeper-0                         1/1     Running     0          46h    10.97.20.9     qa-node018.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-bookkeeper-1                         1/1     Running     0          46h    10.97.20.16    qa-node018.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-bookkeeper-2                         1/1     Running     0          46h    10.97.20.17    qa-node018.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-broker-5847fd6bcf-pjcvj              1/1     Running     19         46h    10.97.20.4     qa-node018.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-proxy-74f48cd977-2kzmd               2/2     Running     0          46h    10.97.20.6     qa-node018.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-zookeeper-0                          1/1     Running     0          46h    10.97.3.105    qa-node001.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-zookeeper-1                          1/1     Running     0          46h    10.97.3.106    qa-node001.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-zookeeper-2                          1/1     Running     0          46h    10.97.3.107    qa-node001.zilliz.local   <none>           <none>
benchmark-tag-z4nwl-1-pulsar-zookeeper-metadata-2g4s2             0/1     Completed   0          46h    10.97.20.5     qa-node018.zilliz.local   <none>           <none>

client pod: benchmark-tag-z4nwl-958768975

client log:

[2022-01-16 14:47:09,424] [    INFO] -  (locust.stats_logger:733)
[2022-01-16 14:47:09,427] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344429.370608741","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:47:09,428] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344429.370608741","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:47:09,428] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344429.371533492","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:47:09,429] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344429.371533492","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:47:09,430] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344429.373513200","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:47:09,431] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344429.373513200","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:47:09,431] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344429.374531843","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:47:09,432] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344429.374531843","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:52:09,390] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344489.379565676","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:52:09,774] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344489.379565676","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:52:09,776] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344489.427522958","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:52:09,777] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344489.427522958","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:52:09,778] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344729.377610389","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,395] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.381529007","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,396] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.381529007","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,398] [    INFO] -  Name                                                                              # reqs      # fails  |     Avg     Min     Max  Median  |   req/s failures/s (locust.stats_logger:725)
[2022-01-16 14:53:09,398] [    INFO] - ---------------------------------------------------------------------------------------------------------------------------------------------------------------- (locust.stats_logger:727)
[2022-01-16 14:53:09,399] [    INFO] -  grpc get                                                                            3841     0(0.00%)  |   12765    1940   72706   12000  |    0.00    0.00 (locust.stats_logger:730)
[2022-01-16 14:53:09,400] [    INFO] -  grpc load_collection                                                                1905     0(0.00%)  |   10958    1184   32574   11000  |    0.00    0.00 (locust.stats_logger:730)
[2022-01-16 14:53:09,612] [    INFO] -  grpc query                                                                         38716    16(0.04%)  |   15541     854  420014   15000  |    0.00    0.00 (locust.stats_logger:730)
[2022-01-16 14:53:09,613] [    INFO] - ---------------------------------------------------------------------------------------------------------------------------------------------------------------- (locust.stats_logger:731)
[2022-01-16 14:53:09,613] [    INFO] -  Aggregated                                                                         44462    16(0.04%)  |   15105     854  420014   15000  |    0.00    0.00 (locust.stats_logger:732)
[2022-01-16 14:53:09,614] [    INFO] -  (locust.stats_logger:733)
[2022-01-16 14:53:09,616] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.383527387","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,616] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.383527387","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,617] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.384531125","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,618] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.384531125","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,619] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.385509379","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,619] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.385509379","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,620] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.386528038","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,621] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.386528038","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,639] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.387504792","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,639] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.387504792","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,640] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.388514722","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,641] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.388514722","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,642] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.389506164","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,643] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.389506164","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,644] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.390537572","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:53:09,644] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344789.390537572","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:54:09,405] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344849.393578251","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:54:09,406] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344849.393578251","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:54:09,472] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344849.395516858","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:54:09,472] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344849.395516858","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:54:09,473] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344849.396524137","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:54:09,474] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344849.396524137","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:59:09,410] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344909.399543409","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:59:09,411] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344909.399543409","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:59:09,412] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344909.400520970","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)
[2022-01-16 14:59:09,413] [   ERROR] - Error: <_MultiThreadedRendezvous of RPC that terminated with:
    status = StatusCode.DEADLINE_EXCEEDED
    details = "Deadline Exceeded"
    debug_error_string = "{"created":"@1642344909.400520970","description":"Error received from peer ipv4:10.96.234.85:19530","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Deadline Exceeded","grpc_status":4}"
> (pymilvus.decorators:75)

xiaocai2333 commented 2 years ago

Pulsar's broker restarted.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.