search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.67k stars 2.92k forks source link

[Bug]: [benchmark][standalone] Occasional `search` raises error `syncTimestamp Failed` under growing index scene #26979

Closed wangting0128 closed 10 months ago

wangting0128 commented 1 year ago

Is there an existing issue for this?

Environment

- Milvus version: 2.3.0-20230909-3e1479e3
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0.dev81
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: new-stable-master-1694293200 client pod: new-stable-master-1694293200-2313532893

server:

[2023-09-10 01:04:51,456 -  INFO - fouram]: [Base] Deploy initial state: 
I0909 21:04:11.029829     392 request.go:665] Waited for 1.169191103s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/argoproj.io/v1alpha1?timeout=32s
NAME                                                              READY   STATUS      RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
new-stable-mast93200-3-74-4705-etcd-0                             1/1     Running     0               3m15s   10.104.13.112   4am-node16   <none>           <none>
new-stable-mast93200-3-74-4705-milvus-standalone-5f567b6db6qpwf   1/1     Running     0               3m15s   10.104.17.201   4am-node23   <none>           <none>
new-stable-mast93200-3-74-4705-minio-69d9d8cf67-j4t4s             1/1     Running     0               3m15s   10.104.24.145   4am-node29   <none>           <none> (base.py:221)
[2023-09-10 01:04:51,456 -  INFO - fouram]: [Cmd Exe]  kubectl get pods  -n qa-milvus  -o wide | grep -E 'STATUS|new-stable-mast93200-3-74-4705-milvus|new-stable-mast93200-3-74-4705-minio|new-stable-mast93200-3-74-4705-etcd|new-stable-mast93200-3-74-4705-pulsar|new-stable-mast93200-3-74-4705-kafka'  (util_cmd.py:14)
[2023-09-10 01:05:00,623 -  INFO - fouram]: [CliClient] pod details of release(new-stable-mast93200-3-74-4705): 
 I0910 01:04:52.721274     588 request.go:665] Waited for 1.164122422s due to client-side throttling, not priority and fairness, request: GET:https://kubernetes.default.svc.cluster.local/apis/metrics.k8s.io/v1beta1?timeout=32s
NAME                                                              READY   STATUS      RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
new-stable-mast93200-3-74-4705-etcd-0                             1/1     Running     0               4h3m    10.104.13.112   4am-node16   <none>           <none>
new-stable-mast93200-3-74-4705-milvus-standalone-5f567b6db6qpwf   1/1     Running     0               4h3m    10.104.17.201   4am-node23   <none>           <none>
new-stable-mast93200-3-74-4705-minio-69d9d8cf67-j4t4s             1/1     Running     0               4h3m    10.104.24.145   4am-node29   <none>           <none>

client logs:

[2023-09-09 23:05:11,099 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2023-09-09 23:05:11,100 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2023-09-09 23:05:11,100 -  INFO - fouram]: grpc     search                                                                       2442265     0(0.00%) |     45       5    3957     43 |   36.80        0.00 (stats.py:789)
[2023-09-09 23:05:11,100 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2023-09-09 23:05:11,100 -  INFO - fouram]:          Aggregated                                                                   2442265     0(0.00%) |     45       5    3957     43 |   36.80        0.00 (stats.py:789)
[2023-09-09 23:05:11,100 -  INFO - fouram]:  (stats.py:790)
[2023-09-09 23:05:11,101 -  INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2023-09-09 23:05:11,101 -  INFO - fouram]: Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs (stats.py:819)
[2023-09-09 23:05:11,101 -  INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2023-09-09 23:05:11,101 -  INFO - fouram]: grpc     search                                                                                 43     45     47     48     52     58     69     82    190    520   4000 2442265 (stats.py:819)
[2023-09-09 23:05:11,101 -  INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2023-09-09 23:05:11,101 -  INFO - fouram]:          Aggregated                                                                             43     45     47     48     52     58     69     82    190    520   4000 2442265 (stats.py:819)
[2023-09-09 23:05:11,101 -  INFO - fouram]:  (stats.py:820)
[2023-09-09 23:05:16,683 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2023-09-09 23:05:05.101741', 'RPC error': '2023-09-09 23:05:16.683706'}> (decorators.py:108)
[2023-09-09 23:05:16,686 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)> (api_request.py:53)
[2023-09-09 23:05:16,686 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)> (func_check.py:46)
[2023-09-09 23:05:17,272 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2023-09-09 23:05:05.111471', 'RPC error': '2023-09-09 23:05:17.272453'}> (decorators.py:108)
[2023-09-09 23:05:17,273 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)> (api_request.py:53)
[2023-09-09 23:05:17,273 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)> (func_check.py:46)
[2023-09-09 23:05:17,276 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2023-09-09 23:05:05.117859', 'RPC error': '2023-09-09 23:05:17.276081'}> (decorators.py:108)
[2023-09-09 23:05:17,276 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)> (api_request.py:53)
[2023-09-09 23:05:17,276 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)> (func_check.py:46)
[2023-09-09 23:05:17,277 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2023-09-09 23:05:05.098231', 'RPC error': '2023-09-09 23:05:17.277390'}> (decorators.py:108)
[2023-09-09 23:05:17,277 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)> (api_request.py:53)
[2023-09-09 23:05:17,278 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)> (func_check.py:46)
[2023-09-09 23:05:17,278 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2023-09-09 23:05:05.114405', 'RPC error': '2023-09-09 23:05:17.278577'}> (decorators.py:108)
[2023-09-09 23:05:17,279 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)> (api_request.py:53)
[2023-09-09 23:05:17,279 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Search
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search: rpc error: code = DeadlineExceeded desc = context deadline exceeded)> (func_check.py:46)
[2023-09-09 23:05:31,112 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2023-09-09 23:05:31,112 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2023-09-09 23:05:31,113 -  INFO - fouram]: grpc     search                                                                       2442697     5(0.00%) |     45       5   12565     43 |   20.50        0.00 (stats.py:789)
[2023-09-09 23:05:31,113 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2023-09-09 23:05:31,113 -  INFO - fouram]:          Aggregated                                                                   2442697     5(0.00%) |     45       5   12565     43 |   20.50        0.00 (stats.py:789)
[2023-09-09 23:05:31,113 -  INFO - fouram]:  (stats.py:790)
[2023-09-09 23:05:31,114 -  INFO - fouram]: Response time percentiles (approximated) (stats.py:819)
[2023-09-09 23:05:31,114 -  INFO - fouram]: Type     Name                                                                                  50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs (stats.py:819)
[2023-09-09 23:05:31,114 -  INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2023-09-09 23:05:31,114 -  INFO - fouram]: grpc     search                                                                                 43     45     47     48     52     58     69     83    190    900  13000 2442697 (stats.py:819)
[2023-09-09 23:05:31,114 -  INFO - fouram]: --------|--------------------------------------------------------------------------------|--------|------|------|------|------|------|------|------|------|------|------|------ (stats.py:819)
[2023-09-09 23:05:31,115 -  INFO - fouram]:          Aggregated                                                                             43     45     47     48     52     58     69     83    190    900  13000 2442697 (stats.py:819)
[2023-09-09 23:05:31,115 -  INFO - fouram]:  (stats.py:820)

test result:

{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'standalone',
            'config_name': 'standalone_8c16m',
            'config': {'standalone': {'resources': {'limits': {'cpu': '8.0',
                                                               'memory': '16Gi'},
                                                    'requests': {'cpu': '5.0',
                                                                 'memory': '9Gi'}},
                                      'persistence': {'persistentVolumeClaim': {'storageClass': 'local-path'}}},
                       'cluster': {'enabled': False},
                       'etcd': {'replicaCount': 1,
                                'global': {'storageClass': 'local-path'},
                                'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'minio': {'mode': 'standalone',
                                 'persistence': {'storageClass': 'local-path'},
                                 'metrics': {'podMonitor': {'enabled': True}}},
                       'pulsar': {'enabled': False},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'nodeSelector': {'node-role/nvme': 'true'},
                       'extraConfigFiles': {'user.yaml': 'dataCoord:\n'
                                                         '  segment:\n'
                                                         '    sealProportion: '
                                                         '0.9\n'
                                                         '    maxIdleTime: '
                                                         '7200\n'
                                                         'queryNode:\n'
                                                         '  segcore:\n'
                                                         '    growing:\n'
                                                         '      enableIndex: '
                                                         'true\n'
                                                         '\n'},
                       'image': {'all': {'repository': 'harbor.milvus.io/dockerhub/milvusdb/milvus',
                                         'tag': '2.3.0-20230909-3e1479e3'}}},
            'host': 'new-stable-mast93200-3-74-4705-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 400000,
                                                    'ni_per': 50000},
                                 'collection_params': {'other_fields': ['float_1'],
                                                       'shards_num': 2},
                                 'load_params': {},
                                 'flush_params': {'prepare_flush': False},
                                 'query_params': {},
                                 'search_params': {},
                                 'resource_groups_params': {'reset': False},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 30,
                                                                  'efConstruction': 360}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '4h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 20,
                                                       'params': {'nq': 10,
                                                                  'top_k': 10,
                                                                  'search_param': {'ef': 69},
                                                                  'expr': {'float_1': {'GT': -1,
                                                                                       'LT': 200000}},
                                                                  'guarantee_timestamp': None,
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'timeout': 60,
                                                                  'random_data': True}}]},
            'run_id': 20230909,
            'datetime': '2023-09-09 21:01:02.097562',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 0.5134},
                            'insert': {'total_time': 12.6048,
                                       'VPS': 31733.9426,
                                       'batch_time': 1.5756,
                                       'batch': 50000},
                            'load': {'RT': 5.0192},
                            'Locust': {'Aggregated': {'Requests': 2891051,
                                                      'Fails': 5,
                                                      'RPS': 200.77,
                                                      'fail_s': 0.0,
                                                      'RT_max': 12565.24,
                                                      'RT_avg': 86.88,
                                                      'TP50': 44,
                                                      'TP99': 390.0},
                                       'search': {'Requests': 2891051,
                                                  'Fails': 5,
                                                  'RPS': 200.77,
                                                  'fail_s': 0.0,
                                                  'RT_max': 12565.24,
                                                  'RT_avg': 86.88,
                                                  'TP50': 44,
                                                  'TP99': 390.0}}}}}

Expected Behavior

No response

Steps To Reproduce

1. Deploy a standalone Milvus and setting queryNode.segcore.growing.enableIndex = true
2. prepare 400k data
3. concurrent search <- raise error

Milvus Log

Milvus log:

(no unique labels) | [2023/09/09 23:05:17.268 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] [traceID=861da6b1c4bc2dd78a6c0a1efe0af055] ["retry time"=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:322\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:406\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321\ngithub.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61\ngithub.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83\ngithub.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170\ngithub.com/milvus-io/milvus/internal/proxy.(*Proxy).Search\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler.func1\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.3.1-0.20230905091144-d8ce91954095/milvuspb/milvus.pb.go:11896\ngithub.com/milvus-io/milvus/internal/proxy.KeepActiveInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/keep_active_interceptor.go:44\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/accesslog/util.go:43\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.RateLimitInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/rate_limit_interceptor.go:48\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/privilege_interceptor.go:61\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerHookInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/hook_interceptor.go:113\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.DatabaseInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/database_interceptor.go:15\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/auth/auth.go:47\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.38.0/interceptor.go:342\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.3.1-0.20230905091144-d8ce91954095/milvuspb/milvus.pb.go:11898\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"]Show context
-- | --
  |   | (no unique labels) | [2023/09/09 23:05:17.268 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] [traceID=321651183fcfd3a05cd761f0ce1b5f4f] ["retry time"=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:322\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:406\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321\ngithub.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61\ngithub.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83\ngithub.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170\ngithub.com/milvus-io/milvus/internal/proxy.(*Proxy).Search\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler.func1\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.3.1-0.20230905091144-d8ce91954095/milvuspb/milvus.pb.go:11896\ngithub.com/milvus-io/milvus/internal/proxy.KeepActiveInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/keep_active_interceptor.go:44\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/accesslog/util.go:43\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.RateLimitInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/rate_limit_interceptor.go:48\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/privilege_interceptor.go:61\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerHookInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/hook_interceptor.go:113\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.DatabaseInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/database_interceptor.go:15\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/auth/auth.go:47\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.38.0/interceptor.go:342\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.3.1-0.20230905091144-d8ce91954095/milvuspb/milvus.pb.go:11898\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"]
  |   | (no unique labels) | [2023/09/09 23:05:17.268 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] [traceID=2a0892fe510bcbb99d8e88bd47608f6c] ["retry time"=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:322\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:406\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321\ngithub.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61\ngithub.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83\ngithub.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170\ngithub.com/milvus-io/milvus/internal/proxy.(*Proxy).Search\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler.func1\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.3.1-0.20230905091144-d8ce91954095/milvuspb/milvus.pb.go:11896\ngithub.com/milvus-io/milvus/internal/proxy.KeepActiveInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/keep_active_interceptor.go:44\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/accesslog/util.go:43\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.RateLimitInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/rate_limit_interceptor.go:48\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/privilege_interceptor.go:61\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerHookInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/hook_interceptor.go:113\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.DatabaseInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/database_interceptor.go:15\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/auth/auth.go:47\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.38.0/interceptor.go:342\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.3.1-0.20230905091144-d8ce91954095/milvuspb/milvus.pb.go:11898\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"]
  |   | (no unique labels) | [2023/09/09 23:05:17.268 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] [traceID=01a18846b7f7ab2a0dfba617e8fa79ad] ["retry time"=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:322\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:406\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321\ngithub.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61\ngithub.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83\ngithub.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170\ngithub.com/milvus-io/milvus/internal/proxy.(*Proxy).Search\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler.func1\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.3.1-0.20230905091144-d8ce91954095/milvuspb/milvus.pb.go:11896\ngithub.com/milvus-io/milvus/internal/proxy.KeepActiveInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/keep_active_interceptor.go:44\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/accesslog/util.go:43\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.RateLimitInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/rate_limit_interceptor.go:48\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/privilege_interceptor.go:61\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerHookInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/hook_interceptor.go:113\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.DatabaseInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/database_interceptor.go:15\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/auth/auth.go:47\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.38.0/interceptor.go:342\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.3.1-0.20230905091144-d8ce91954095/milvuspb/milvus.pb.go:11898\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"]
  |   | (no unique labels) | [2023/09/09 23:05:16.673 +00:00] [ERROR] [retry/retry.go:42] ["retry func failed"] [traceID=2f6e6c5a9964a228303c7fa3254a34f9] ["retry time"=0] [error="rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/go/src/github.com/milvus-io/milvus/pkg/util/retry/retry.go:42\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:322\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:406\ngithub.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n\t/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120\ngithub.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321\ngithub.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61\ngithub.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83\ngithub.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170\ngithub.com/milvus-io/milvus/internal/proxy.(*Proxy).Search\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2493\ngithub.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Search\n\t/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:830\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler.func1\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.3.1-0.20230905091144-d8ce91954095/milvuspb/milvus.pb.go:11896\ngithub.com/milvus-io/milvus/internal/proxy.KeepActiveInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/keep_active_interceptor.go:44\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy/accesslog.UnaryAccessLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/accesslog/util.go:43\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.RateLimitInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/rate_limit_interceptor.go:48\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n\t/go/src/github.com/milvus-io/milvus/pkg/util/logutil/grpc_interceptor.go:23\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/privilege_interceptor.go:61\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.UnaryServerHookInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/hook_interceptor.go:113\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/milvus-io/milvus/internal/proxy.DatabaseInterceptor.func1\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/database_interceptor.go:15\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/auth/auth.go:47\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.38.0/interceptor.go:342\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25\ngithub.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1\n\t/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34\ngithub.com/milvus-io/milvus-proto/go-api/v2/milvuspb._MilvusService_Search_Handler\n\t/go/pkg/mod/github.com/milvus-io/milvus-proto/go-api/v2@v2.3.1-0.20230905091144-d8ce91954095/milvuspb/milvus.pb.go:11898\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"]
  |   | (no unique labels)

Anything else?

No response

xiaofan-luan commented 1 year ago

/assign @chyezh could you help on investigating this?

yanliang567 commented 1 year ago

/unassign

chyezh commented 1 year ago

A timeout (context.DeadlineExceed) during the search process cause that the Proxy component to perform a GRPC reconnection to local rootCoord. The error was recovered by one seconds. There's no more log context on rootCoord to find the cause of the problem. It may be just a network error. image A merged PR https://github.com/milvus-io/milvus/pull/27101 has been optimized the reconnection logic. I think this issue can be closed and wait for reproducing, then reopen. @wangting0128

wangting0128 commented 1 year ago

A timeout (context.DeadlineExceed) during the search process cause that the Proxy component to perform a GRPC reconnection to local rootCoord. The error was recovered by one seconds. There's no more log context on rootCoord to find the cause of the problem. It may be just a network error. image A merged PR #27101 has been optimized the reconnection logic. I think this issue can be closed and wait for reproducing, then reopen. @wangting0128

@elstic Please follow up on this issue

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

stale[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.