search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.07k stars 2.88k forks source link

[Bug]: Search failed with error `reason=Query 23 failed, reason context canceled err` after reinstallation #21662

Closed zhuwenxing closed 1 year ago

zhuwenxing commented 1 year ago

Is there an existing issue for this?

Environment

- Milvus version:master-20230111-9f809599
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):kafka    
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus==2.3.0.dev21
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2023-01-11T10:40:14.140Z] <name>: deploy_test_index_type_HNSW_is_compacted_is_compacted_segment_status_all_is_string_indexed_is_string_indexed_replica_number_0_is_deleted_is_deleted_data_size_3000

[2023-01-11T10:40:14.141Z] <partitions>: [{"name": "_default", "collection_name": "deploy_test_index_type_HNSW_is_compacted_is_c......  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:57 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 120} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:57 - DEBUG - ci_test]: (api_response) : None  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:57 - INFO - ci_test]: inserted 3000 data into collection deploy_test_index_type_HNSW_is_compacted_is_compacted_segment_status_all_is_string_indexed_is_string_indexed_replica_number_0_is_deleted_is_deleted_data_size_3000 (common_func.py:740)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:57 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [      int64   float varchar                                       float_vector

[2023-01-11T10:40:14.141Z] 0         0     0.0       0  [0.08677974339739154, 0.020162897897669098, 0....

[2023-01-11T10:40:14.141Z] 1         1     1.0       1  [0.08827132732353106, 0.11569094861651397, 0.1...

[2023-01-11T10:40:14.141Z] 2         2     2.0       2  [0.12329915870123, 0.084804579739......, kwargs: {'timeout': 120} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_response) : (insert count: 3000, delete count: 0, upsert count: 0, timestamp: 438680545646346241, success count: 3000, err count: 0)  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_request)  : [Connections.connect] args: ['default'], kwargs: {'host': '10.101.59.69', 'port': 19530} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_response) : None  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['int64', <DataType.INT64: 5>, ''], kwargs: {'is_primary': False} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_response) : {

[2023-01-11T10:40:14.141Z]     name: int64

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 5

[2023-01-11T10:40:14.141Z]   }  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['float', <DataType.FLOAT: 10>, ''], kwargs: {'is_primary': False} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_response) : {

[2023-01-11T10:40:14.141Z]     name: float

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 10

[2023-01-11T10:40:14.141Z]   }  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['varchar', <DataType.VARCHAR: 21>, ''], kwargs: {'max_length': 65535, 'is_primary': False} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_response) : {

[2023-01-11T10:40:14.141Z]     name: varchar

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 21

[2023-01-11T10:40:14.141Z]     params: {'max_length': 65535}

[2023-01-11T10:40:14.141Z]   }  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_request)  : [FieldSchema] args: ['float_vector', <DataType.FLOAT_VECTOR: 101>, ''], kwargs: {'dim': 128, 'is_primary': False} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_response) : {

[2023-01-11T10:40:14.141Z]     name: float_vector

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 101

[2023-01-11T10:40:14.141Z]     params: {'dim': 128}

[2023-01-11T10:40:14.141Z]   }  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_request)  : [CollectionSchema] args: [[{

[2023-01-11T10:40:14.141Z]     name: int64

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 5

[2023-01-11T10:40:14.141Z]   }, {

[2023-01-11T10:40:14.141Z]     name: float

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 10

[2023-01-11T10:40:14.141Z]   }, {

[2023-01-11T10:40:14.141Z]     name: varchar

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 21

[2023-01-11T10:40:14.141Z]     params: {'max_length': 65535}

[2023-01-11T10:40:14.141Z]   }, {

[2023-01-11T10:40:14.141Z]     name: float_vector

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 101

[2023-01-11T10:40:14.141Z]     params: {'dim': 128}

[2023-01-11T10:40:14.141Z]   }], ''], kwargs: {'primary_field': 'int64', 'auto_id': False} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_response) : {

[2023-01-11T10:40:14.141Z]   auto_id: False

[2023-01-11T10:40:14.141Z]   description: 

[2023-01-11T10:40:14.141Z]   fields: [{

[2023-01-11T10:40:14.141Z]     name: int64

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 5

[2023-01-11T10:40:14.141Z]     is_primary: True

[2023-01-11T10:40:14.141Z]     auto_id: False

[2023-01-11T10:40:14.141Z]   }, {

[2023-01-11T10:40:14.141Z]     name: float

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 10

[2023-01-11T10:40:14.141Z]   }, {

[2023-01-11T10:40:14.141Z]     name: varchar

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 21

[2023-01-11T10:40:14.141Z]     params: {'max_length': 65535}

[2023-01-11T10:40:14.141Z]   }, {

[2023-01-11T10:40:14.141Z]     name: float......  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - INFO - ci_test]: init_collection_general: collection creation (client_base.py:157)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_request)  : [Connections.has_connection] args: ['default'], kwargs: {} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_response) : True  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_request)  : [Collection] args: ['deploy_test_index_type_HNSW_is_compacted_is_compacted_segment_status_all_is_string_indexed_is_string_indexed_replica_number_0_is_deleted_is_deleted_data_size_3000', {

[2023-01-11T10:40:14.141Z]   auto_id: False

[2023-01-11T10:40:14.141Z]   description: 

[2023-01-11T10:40:14.141Z]   fields: [{

[2023-01-11T10:40:14.141Z]     name: int64

[2023-01-11T10:40:14.141Z]     description: 

[2023-01-11T10:40:14.141Z]     type: 5

[2023-01-11T10:40:14.141Z]     is_primary: True

[2023-01-11T10:40:14.141Z]     auto_id: False......, kwargs: {'is_index': True, 'consistency_level': 'Strong'} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_response) : <Collection>:

[2023-01-11T10:40:14.141Z] -------------

[2023-01-11T10:40:14.141Z] <name>: deploy_test_index_type_HNSW_is_compacted_is_compacted_segment_status_all_is_string_indexed_is_string_indexed_replica_number_0_is_deleted_is_deleted_data_size_3000

[2023-01-11T10:40:14.141Z] <partitions>: [{"name": "_default", "collection_name": "deploy_test_index_type_HNSW_is_compacted_is_c......  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:37:58 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 120} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:38:00 - DEBUG - ci_test]: (api_response) : None  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:38:00 - INFO - ci_test]: inserted 3000 data into collection deploy_test_index_type_HNSW_is_compacted_is_compacted_segment_status_all_is_string_indexed_is_string_indexed_replica_number_0_is_deleted_is_deleted_data_size_3000 (common_func.py:740)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:38:00 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [      int64   float varchar                                       float_vector

[2023-01-11T10:40:14.141Z] 0         0     0.0       0  [0.030125711882818592, 0.1448238625838024, 0.0...

[2023-01-11T10:40:14.141Z] 1         1     1.0       1  [0.12383164889993978, 0.03064027915123323, 0.1...

[2023-01-11T10:40:14.141Z] 2         2     2.0       2  [0.07585493719005834, 0.071230977......, kwargs: {'timeout': 120} (api_request.py:56)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:38:00 - DEBUG - ci_test]: (api_response) : (insert count: 3000, delete count: 0, upsert count: 0, timestamp: 438680546367242242, success count: 3000, err count: 0)  (api_request.py:31)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:38:10 - INFO - ci_test]: index info: [] (test_action_second_deployment.py:58)

[2023-01-11T10:40:14.141Z] [2023-01-11 10:38:40 - ERROR - pymilvus.decorators]: RPC error: [query], <MilvusException: (code=1, message=fail to query on all shard leaders, err=fail to Query, QueryNode ID = 23, reason=Query 23 failed, reason context canceled err %!w(<nil>))>, <Time:{'RPC start': '2023-01-11 10:38:40.179068', 'RPC error': '2023-01-11 10:38:40.381457'}> (decorators.py:108)
[2023-01-11T10:40:14.141Z] FAILED testcases/test_action_second_deployment.py::TestActionSecondDeployment::test_check[deploy_test_index_type_BIN_IVF_FLAT_is_compacted_is_compacted_segment_status_all_is_string_indexed_is_string_indexed_replica_number_0_is_deleted_is_deleted_data_size_3000] - pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=23, reason=Search 17 failed, reason Timestamp lag too large lag(464842h33m32.812s) max(24h0m0s) err %!w(<nil>))>

[2023-01-11T10:40:14.141Z] FAILED testcases/test_action_second_deployment.py::TestActionSecondDeployment::test_check[deploy_test_index_type_HNSW_is_compacted_is_compacted_segment_status_all_is_string_indexed_is_string_indexed_replica_number_0_is_deleted_is_deleted_data_size_3000] - pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to query on all shard leaders, err=fail to Query, QueryNode ID = 23, reason=Query 23 failed, reason context canceled err %!w(<nil>))>

[2023-01-11T10:40:14.141Z] =================== 2 failed, 48 passed in 523.56s (0:08:43) ===================

Expected Behavior

all test cases passed

Steps To Reproduce

No response

Milvus Log

deploy task: reinstall old image tag: v2.2.0 new image tag: master-20230111-9f809599 failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_kafka_cron/detail/deploy_test_kafka_cron/202/pipeline

log: artifacts-kafka-cluster-reinstall-202-server-second-deployment-logs.tar.gz artifacts-kafka-cluster-reinstall-202-server-first-deployment-logs.tar.gz artifacts-kafka-cluster-reinstall-202-pytest-logs.tar.gz

Anything else?

No response

binbinlv commented 1 year ago

@congqixia

could you please have a look first? Thanks.

zhuwenxing commented 1 year ago

It also reproduced in 2.2.0-20230117-1a819f2d

[2023-01-17T13:01:02.428Z] <name>: deploy_test_index_type_HNSW_is_compacted_not_compacted_segment_status_only_growing_is_string_indexed_not_string_indexed_replica_number_2_is_deleted_is_deleted_data_size_3000

[2023-01-17T13:01:02.428Z] <partitions>: [{"name": "_default", "collection_name": "deploy_test_index_type_HNSW_is_com......  (api_request.py:31)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:54:49 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 120} (api_request.py:56)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:54:53 - DEBUG - ci_test]: (api_response) : None  (api_request.py:31)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:54:53 - INFO - ci_test]: inserted 3000 data into collection deploy_test_index_type_HNSW_is_compacted_not_compacted_segment_status_only_growing_is_string_indexed_not_string_indexed_replica_number_2_is_deleted_is_deleted_data_size_3000 (common_func.py:740)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:54:53 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [      int64   float varchar                                       float_vector

[2023-01-17T13:01:02.428Z] 0         0     0.0       0  [0.06622585469149442, 0.05149480071923069, 0.0...

[2023-01-17T13:01:02.428Z] 1         1     1.0       1  [0.009027094743335583, 0.14389109021241628, 0....

[2023-01-17T13:01:02.428Z] 2         2     2.0       2  [0.003013331986805464, 0.15043796......, kwargs: {'timeout': 120} (api_request.py:56)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:54:53 - DEBUG - ci_test]: (api_response) : (insert count: 3000, delete count: 0, upsert count: 0, timestamp: 438818594807611393, success count: 3000, err count: 0)  (api_request.py:31)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:55:01 - INFO - ci_test]: index info: [] (test_action_second_deployment.py:58)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:55:09 - ERROR - pymilvus.decorators]: RPC error: [query], <MilvusException: (code=1, message=fail to query on all shard leaders, err=fail to Query, QueryNode ID = 18, reason=Query 19 failed, reason  err err: rpc error: code = Canceled desc = context canceled

[2023-01-17T13:01:02.428Z] , /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace

[2023-01-17T13:01:02.429Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:277 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call

[2023-01-17T13:01:02.429Z] /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:268 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Query

[2023-01-17T13:01:02.429Z] /go/src/github.com/milvus-io/milvus/internal/querynode/shard_cluster.go:1076 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Query.func2

[2023-01-17T13:01:02.429Z] /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit

[2023-01-17T13:01:02.429Z] )>, <Time:{'RPC start': '2023-01-17 12:55:08.788101', 'RPC error': '2023-01-17 12:55:09.117405'}> (decorators.py:108)

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_kafka_for_release_cron/detail/deploy_test_kafka_for_release_cron/228/pipeline log:

artifacts-kafka-cluster-reinstall-228-server-second-deployment-logs.tar.gz artifacts-kafka-cluster-reinstall-228-server-first-deployment-logs.tar.gz artifacts-kafka-cluster-reinstall-228-pytest-logs.tar.gz

zhuwenxing commented 1 year ago

It also reproduced in 2.2.0-20230117-1a819f2d

[2023-01-17T13:01:02.428Z] <name>: deploy_test_index_type_HNSW_is_compacted_not_compacted_segment_status_only_growing_is_string_indexed_not_string_indexed_replica_number_2_is_deleted_is_deleted_data_size_3000

[2023-01-17T13:01:02.428Z] <partitions>: [{"name": "_default", "collection_name": "deploy_test_index_type_HNSW_is_com......  (api_request.py:31)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:54:49 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 120} (api_request.py:56)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:54:53 - DEBUG - ci_test]: (api_response) : None  (api_request.py:31)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:54:53 - INFO - ci_test]: inserted 3000 data into collection deploy_test_index_type_HNSW_is_compacted_not_compacted_segment_status_only_growing_is_string_indexed_not_string_indexed_replica_number_2_is_deleted_is_deleted_data_size_3000 (common_func.py:740)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:54:53 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [      int64   float varchar                                       float_vector

[2023-01-17T13:01:02.428Z] 0         0     0.0       0  [0.06622585469149442, 0.05149480071923069, 0.0...

[2023-01-17T13:01:02.428Z] 1         1     1.0       1  [0.009027094743335583, 0.14389109021241628, 0....

[2023-01-17T13:01:02.428Z] 2         2     2.0       2  [0.003013331986805464, 0.15043796......, kwargs: {'timeout': 120} (api_request.py:56)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:54:53 - DEBUG - ci_test]: (api_response) : (insert count: 3000, delete count: 0, upsert count: 0, timestamp: 438818594807611393, success count: 3000, err count: 0)  (api_request.py:31)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:55:01 - INFO - ci_test]: index info: [] (test_action_second_deployment.py:58)

[2023-01-17T13:01:02.428Z] [2023-01-17 12:55:09 - ERROR - pymilvus.decorators]: RPC error: [query], <MilvusException: (code=1, message=fail to query on all shard leaders, err=fail to Query, QueryNode ID = 18, reason=Query 19 failed, reason  err err: rpc error: code = Canceled desc = context canceled

[2023-01-17T13:01:02.428Z] , /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace

[2023-01-17T13:01:02.429Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:277 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call

[2023-01-17T13:01:02.429Z] /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:268 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Query

[2023-01-17T13:01:02.429Z] /go/src/github.com/milvus-io/milvus/internal/querynode/shard_cluster.go:1076 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Query.func2

[2023-01-17T13:01:02.429Z] /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit

[2023-01-17T13:01:02.429Z] )>, <Time:{'RPC start': '2023-01-17 12:55:08.788101', 'RPC error': '2023-01-17 12:55:09.117405'}> (decorators.py:108)

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_kafka_for_release_cron/detail/deploy_test_kafka_for_release_cron/228/pipeline log:

artifacts-kafka-cluster-reinstall-228-server-second-deployment-logs.tar.gz artifacts-kafka-cluster-reinstall-228-server-first-deployment-logs.tar.gz artifacts-kafka-cluster-reinstall-228-pytest-logs.tar.gz

The error is raised by the query

[2023-01-17T13:01:02.427Z]     @retry_on_rpc_failure()

[2023-01-17T13:01:02.427Z]     def query(self, collection_name, expr, output_fields=None, partition_names=None, timeout=None, **kwargs):

[2023-01-17T13:01:02.427Z]         if output_fields is not None and not isinstance(output_fields, (list,)):

[2023-01-17T13:01:02.427Z]             raise ParamError(message="Invalid query format. 'output_fields' must be a list")

[2023-01-17T13:01:02.427Z]         collection_schema = kwargs.get("schema", None)

[2023-01-17T13:01:02.427Z]         if not collection_schema:

[2023-01-17T13:01:02.427Z]             collection_schema = self.describe_collection(collection_name, timeout)

[2023-01-17T13:01:02.427Z]         consistency_level = collection_schema["consistency_level"]

[2023-01-17T13:01:02.427Z]         # overwrite the consistency level defined when user created the collection

[2023-01-17T13:01:02.427Z]         consistency_level = get_consistency_level(kwargs.get("consistency_level", consistency_level))

[2023-01-17T13:01:02.427Z]     

[2023-01-17T13:01:02.427Z]         ts_utils.construct_guarantee_ts(consistency_level, collection_name, kwargs)

[2023-01-17T13:01:02.427Z]         request = Prepare.query_request(collection_name, expr, output_fields, partition_names, **kwargs)

[2023-01-17T13:01:02.427Z]     

[2023-01-17T13:01:02.427Z]         future = self._stub.Query.future(request, timeout=timeout)

[2023-01-17T13:01:02.427Z]         response = future.result()

[2023-01-17T13:01:02.427Z]         if response.status.error_code == Status.EMPTY_COLLECTION:

[2023-01-17T13:01:02.427Z]             return []

[2023-01-17T13:01:02.427Z]         if response.status.error_code != Status.SUCCESS:

[2023-01-17T13:01:02.427Z] >           raise MilvusException(response.status.error_code, response.status.reason)

[2023-01-17T13:01:02.427Z] E           pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to query on all shard leaders, err=fail to Query, QueryNode ID = 18, reason=Query 19 failed, reason  err err: rpc error: code = Canceled desc = context canceled

[2023-01-17T13:01:02.427Z] E           , /go/src/github.com/milvus-io/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace

[2023-01-17T13:01:02.427Z] E           /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:277 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call

[2023-01-17T13:01:02.427Z] E           /go/src/github.com/milvus-io/milvus/internal/distributed/querynode/client/client.go:268 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).Query

[2023-01-17T13:01:02.427Z] E           /go/src/github.com/milvus-io/milvus/internal/querynode/shard_cluster.go:1076 github.com/milvus-io/milvus/internal/querynode.(*ShardCluster).Query.func2

[2023-01-17T13:01:02.427Z] E           /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit

[2023-01-17T13:01:02.427Z] E           )>

[2023-01-17T13:01:02.427Z] 

[2023-01-17T13:01:02.427Z] /usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py:885: MilvusException
stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.