[Bug]: Milvus search exception

[HuaJieHappy]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:2.2.12
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):  defalut  
- SDK version(e.g. pymilvus v2.0.0rc2): 2.2.12
- OS(Ubuntu or CentOS): centos7.5
- CPU/Memory: 8C32G
- GPU: No
- Others:

Current Behavior

The application was subjected to stress testing with 30 concurrent users, and an error occurred during the query method. The error message is The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/chatfile/app/service/chat_sse_support/multiple_recall.py", line 41, in multiple_recall milvus_docs, milvus_image_ids = milvus_search_based_service(data, question_embedding, search_limit, File "/chatfile/app/recall_pics/milvus_recall.py", line 167, in milvus_search_based_service result = search_vectors(collection=collection, vectors_to_search=content_embedding, search_params=search_params, File "/chatfile/app/service/chat/milvus_search.py", line 19, in search_vectors results = collection.search(data=[vectors_to_search], param=search_params, limit=milvus_search_limit, File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 629, in search res = conn.search(self._name, data, anns_field, param, limit, expr, File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler raise e File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler return func(*args, *kwargs) File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler ret = func(self, args, **kwargs) File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 56, in handler raise MilvusException(message=str(e)) from e pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=<_InactiveRpcError of RPC that terminated with: status = StatusCode.CANCELLED details = "Channel closed!" debug_error_string = "UNKNOWN:Channel closed! {created_time:"2024-10-09T19:01:25.108709223+08:00", grpc_message:"Channel closed!", grpc_status:1}"

)>

Expected Behavior

Is it necessary to modify the configuration or upgrade the version

Steps To Reproduce

use milvus default config

Milvus Log

No response

Anything else?

No response

[yanliang567]

@HuaJieHappy Please refer this doc to export the whole Milvus logs for investigation. For Milvus installed with docker-compose, you can use docker-compose logs > milvus.log to export the logs.

Also quick questions:

1. do you have writing requests during the stress testing?
2. any metrics about the resource usage and milvus pods? if convenient, I suggest you upgrade to 2.3.22 or 2.4.13 as the running milvus 2.2.12 is quite old.

/assign @HuaJieHappy /unassign

[HuaJieHappy]

do we have complete upgrade plan such as from 2.2.12 to 2.4.13? data can't lost

[HuaJieHappy]

How to upgrade Milvus from version 2.2.12 to 2.4.13 or another version while preserving the data and ensuring that queries can still be executed?