search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.75k stars 2.93k forks source link

[Bug]: [Nightly] Load collection will occasionally fail for timeout #31522

Closed NicoYuan1986 closed 1 month ago

NicoYuan1986 commented 8 months ago

Is there an existing issue for this?

Environment

- Milvus version: 2.3-20240321-f8496dbc-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Load collection will occasionally fail for timeout.

[2024-03-21T23:23:40.704Z] [2024-03-21 22:15:15 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-03-21T23:23:40.704Z] [2024-03-21 22:15:20 - ERROR - pymilvus.decorators]: RPC error: [load_collection], <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:275 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ShowPartitions
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:97 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetPartitions
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:118 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue
[2024-03-21T23:23:40.705Z] /usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2024-03-21 22:15:15.452549', 'RPC error': '2024-03-21 22:15:20.464219'}> (decorators.py:139)
[2024-03-21T23:23:40.705Z] [2024-03-21 22:15:20 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-21T23:23:40.705Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-03-21T23:23:40.705Z]     res = func(*args, **_kwargs)
[2024-03-21T23:23:40.705Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-03-21T23:23:40.705Z]     return func(*arg, **kwargs)
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 419, in load
[2024-03-21T23:23:40.705Z]     conn.load_collection(
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-03-21T23:23:40.705Z]     raise e from e
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-03-21T23:23:40.705Z]     return func(*args, **kwargs)
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-03-21T23:23:40.705Z]     return func(self, *args, **kwargs)
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 115, in handler
[2024-03-21T23:23:40.705Z]     raise e from e
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 86, in handler
[2024-03-21T23:23:40.705Z]     return func(*args, **kwargs)
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1046, in load_collection
[2024-03-21T23:23:40.705Z]     check_status(response)
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/utils.py", line 54, in check_status
[2024-03-21T23:23:40.705Z]     raise MilvusException(status.code, status.reason, status.error_code)
[2024-03-21T23:23:40.705Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:275 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ShowPartitions
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:97 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetPartitions
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:118 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue
[2024-03-21T23:23:40.705Z] /usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>
[2024-03-21T23:23:40.705Z]  (api_request.py:45)
[2024-03-21T23:23:40.705Z] [2024-03-21 22:15:20 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*Cli...... (api_request.py:46)[create_path] folder(/tmp/ci_logs/test) is not exist.

Expected Behavior

load successfully

Steps To Reproduce

No response

Milvus Log

  1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/140/pipeline/205/
  2. log: artifacts-milvus-distributed-kafka-nightly-140-pymilvus-e2e-logs.tar.gz
  3. collection name: search_collection_So5wlFEE
  4. failed time: [2024-03-21T22:15:22.118Z] [gw4] [ 49%] FAILED testcases/test_search.py::TestCollectionSearch::test_search_after_different_index_with_params[128-True-True-True-IVF_SQ8-params2]

Anything else?

No response

yanliang567 commented 8 months ago

/assign @weiliu1031 /unassign

weiliu1031 commented 7 months ago

should be fixed by #31798, please verify this with latest image

weiliu1031 commented 7 months ago

/assign @NicoYuan1986

NicoYuan1986 commented 7 months ago

The error has not occurred since the pr merged. So close the issue. I will reopen it if the issue reproduces.

NicoYuan1986 commented 7 months ago

reproduced on 2.3 branch.

  1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/171/pipeline/237
  2. log: artifacts-milvus-distributed-pulsar-nightly-171-pymilvus-e2e-logs.tar.gz
  3. collection name: search_collection_VKAMMKtC
  4. failed time: [2024-04-17T23:04:45.481Z] [gw0] [ 66%] FAILED testcases/test_search.py::TestCollectionSearch::test_search_with_consistency_bounded[200-128-True-True-True]
  5. error message: 
    [2024-04-17T23:44:54.981Z] [2024-04-17 23:04:39 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:44 - ERROR - pymilvus.decorators]: RPC error: [load_collection], <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:275 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ShowPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:97 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:118 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue
[2024-04-17T23:44:54.981Z] /usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2024-04-17 23:04:39.053386', 'RPC error': '2024-04-17 23:04:44.068447'}> (decorators.py:146)
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:44 - ERROR - ci_test]: Traceback (most recent call last):
[2024-04-17T23:44:54.981Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-04-17T23:44:54.981Z]     res = func(*args, **_kwargs)
[2024-04-17T23:44:54.981Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-04-17T23:44:54.981Z]     return func(*arg, **kwargs)
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 419, in load
[2024-04-17T23:44:54.981Z]     conn.load_collection(
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 147, in handler
[2024-04-17T23:44:54.981Z]     raise e from e
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 143, in handler
[2024-04-17T23:44:54.981Z]     return func(*args, **kwargs)
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 182, in handler
[2024-04-17T23:44:54.981Z]     return func(self, *args, **kwargs)
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 122, in handler
[2024-04-17T23:44:54.981Z]     raise e from e
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 87, in handler
[2024-04-17T23:44:54.981Z]     return func(*args, **kwargs)
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1046, in load_collection
[2024-04-17T23:44:54.981Z]     check_status(response)
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/utils.py", line 54, in check_status
[2024-04-17T23:44:54.981Z]     raise MilvusException(status.code, status.reason, status.error_code)
[2024-04-17T23:44:54.981Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:275 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ShowPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:97 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:118 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue
[2024-04-17T23:44:54.981Z] /usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>
[2024-04-17T23:44:54.981Z]  (api_request.py:45)
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:44 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*Cli...... (api_request.py:46)
stale[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

yanliang567 commented 6 months ago

keep it as i was told that it reproduces recently again.

NicoYuan1986 commented 6 months ago

update:

  1. link: https://qa-jenkins.milvus.io/blue/organizations/jenkins/E2E%20Test/detail/E2E%20Test/468/pipeline/
  2. log: artifacts-e2e-test-468-server-logs.tar.gz
  3. collection name: delete_luiZAzlB
  4. failed time: [2024-05-17T10:32:41.809Z] [gw0] [ 16%] FAILED test_delete.py::TestDeleteComplexExpr::test_delete_string_expressions_normal[True-NewStr not in ["0", "1", "2"]]
  5. error message: 
    [2024-05-17T11:09:13.732Z] [2024-05-17 10:32:28 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-05-17T11:09:13.733Z] [2024-05-17 10:32:36 - ERROR - pymilvus.decorators]: RPC error: [load_collection], <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:556 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:749 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).ListIndexes
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:296 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).ListIndexes
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/utils.go:88 github.com/milvus-io/milvus/internal/querycoordv2/job.loadPartitions
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:176 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2024-05-17 10:32:28.275153', 'RPC error': '2024-05-17 10:32:36.210539'}> (decorators.py:139)
[2024-05-17T11:09:13.733Z] [2024-05-17 10:32:36 - ERROR - ci_test]: Traceback (most recent call last):
[2024-05-17T11:09:13.733Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-05-17T11:09:13.733Z]     res = func(*args, **_kwargs)
[2024-05-17T11:09:13.733Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-05-17T11:09:13.733Z]     return func(*arg, **kwargs)
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 424, in load
[2024-05-17T11:09:13.733Z]     conn.load_collection(
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 140, in handler
[2024-05-17T11:09:13.733Z]     raise e from e
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler
[2024-05-17T11:09:13.733Z]     return func(*args, **kwargs)
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 175, in handler
[2024-05-17T11:09:13.733Z]     return func(self, *args, **kwargs)
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 115, in handler
[2024-05-17T11:09:13.733Z]     raise e from e
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 86, in handler
[2024-05-17T11:09:13.733Z]     return func(*args, **kwargs)
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 1148, in load_collection
[2024-05-17T11:09:13.733Z]     check_status(response)
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/utils.py", line 63, in check_status
[2024-05-17T11:09:13.733Z]     raise MilvusException(status.code, status.reason, status.error_code)
[2024-05-17T11:09:13.733Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:556 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:749 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).ListIndexes
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:296 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).ListIndexes
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/utils.go:88 github.com/milvus-io/milvus/internal/querycoordv2/job.loadPartitions
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:176 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>
[2024-05-17T11:09:13.733Z]  (api_request.py:45)
[2024-05-17T11:09:13.733Z] [2024-05-17 10:32:36 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:556 github.com/milvus-io/milvus/internal/util/grpcclient.(*Cli...... (api_request.py:46)
NicoYuan1986 commented 6 months ago

update:

  1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/205/pipeline/205/
  2. log: artifacts-milvus-distributed-kafka-nightly-205-pymilvus-e2e-logs.tar.gz
  3. collection name: coll__ffQ9yake
  4. failed time: [2024-05-21T21:19:01.431Z] [gw0] [ 23%] FAILED testcases/test_partition.py::TestPartitionParams::test_load_partition_after_load_partition
  5. error message: 
    2024-05-21T23:00:09.199Z] [2024-05-21 21:18:55 - DEBUG - ci_test]: (api_request)  : [Partition.load] args: [1, 180], kwargs: {} (api_request.py:62)
[2024-05-21T23:00:09.199Z] [2024-05-21 21:19:00 - ERROR - pymilvus.decorators]: RPC error: [load_partitions], <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:183 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:78 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetCollectionSchema
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/utils.go:76 github.com/milvus-io/milvus/internal/querycoordv2/job.loadPartitions
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:352 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadPartitionJob).Execute
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2024-05-21 21:18:55.092973', 'RPC error': '2024-05-21 21:19:00.113220'}> (decorators.py:139)
[2024-05-21T23:00:09.199Z] [2024-05-21 21:19:00 - ERROR - ci_test]: Traceback (most recent call last):
[2024-05-21T23:00:09.199Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-05-21T23:00:09.199Z]     res = func(*args, **_kwargs)
[2024-05-21T23:00:09.199Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-05-21T23:00:09.199Z]     return func(*arg, **kwargs)
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/partition.py", line 199, in load
[2024-05-21T23:00:09.199Z]     return conn.load_partitions(
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-05-21T23:00:09.199Z]     raise e from e
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-05-21T23:00:09.199Z]     return func(*args, **kwargs)
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-05-21T23:00:09.199Z]     return func(self, *args, **kwargs)
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 115, in handler
[2024-05-21T23:00:09.199Z]     raise e from e
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 86, in handler
[2024-05-21T23:00:09.199Z]     return func(*args, **kwargs)
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1127, in load_partitions
[2024-05-21T23:00:09.199Z]     check_status(response)
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/utils.py", line 54, in check_status
[2024-05-21T23:00:09.199Z]     raise MilvusException(status.code, status.reason, status.error_code)
[2024-05-21T23:00:09.199Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:183 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:78 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetCollectionSchema
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/utils.go:76 github.com/milvus-io/milvus/internal/querycoordv2/job.loadPartitions
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:352 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadPartitionJob).Execute
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>
[2024-05-21T23:00:09.199Z]  (api_request.py:45)
[2024-05-21T23:00:09.200Z] [2024-05-21 21:19:00 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-21T23:00:09.200Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*Cli...... (api_request.py:46)
NicoYuan1986 commented 5 months ago

update: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/206/pipeline/237

NicoYuan1986 commented 2 months ago

update: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.4/143/pipeline/179/ time: [2024-08-26T20:54:58.404Z] [gw4] [ 49%] FAILED testcases/test_query.py::TestQueryParams::test_mmap_query_string_is_primary[(varchar > "0" && varchar < "100") or (varchar > "200" && varchar < "300")]

NicoYuan1986 commented 1 month ago

not reproduce recently. so close the issue now.