[QUESTION]: [describe_collection] retry:10, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses;

wulongjian commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

What is your question?

Hello, I am a starter in pymilvus. my pymilvus version is 2.4.4

# packages in environment at /opt/conda/envs/image_search:
#
# Name                    Version                   Build  Channel
pymilvus                  2.4.4                    pypi_0    pypi

my code is very sample

from pymilvus import MilvusClient
import numpy as np
import os
import time

client = MilvusClient("image_search_wulj_test.db")#3
if client.has_collection(collection_name="image_embeddings"):
    client.drop_collection(collection_name="image_embeddings")
client.create_collection(
    collection_name="image_embeddings",
    vector_field_name="vector",
    dimension=512,
    auto_id=True,
    enable_dynamic_field=True,
    metric_type="COSINE",
)
#4

cnt = 0
from tqdm import tqdm
import json
import numpy as np
with open('truck_data_resnet34.jsonl','r') as fi:

    for line in fi:
        filepath = '/opt/data/private/shuowang/image_data/image/image/' #整图
        data = json.loads(line)
        filepath += str(data['path']) + ".jpg"
        for item in data['truck']:
            image_embedding = np.array(item['vector'])

            client.insert(
                    "image_embeddings",
                    {"vector": image_embedding, "filename": filepath+item['path']},
                )
        cnt += 1
        # time.sleep(10)
        print('cur cnt: ', cnt)

when i want write embedding infos from a json file to milvus db, it occurs errors: the client.insert() function works fun for the first few minutes, , about several minutes later, it breaks as follows:

[describe_collection] retry:72, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory>
[describe_collection] retry:73, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory>
[describe_collection] retry:74, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory>
[describe_collection] retry:75, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory>
RPC error: [describe_collection], <MilvusException: (code=<bound method _MultiThreadedRendezvous.code of <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-07-30T20:16:16.002968128+08:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory"}"
>>, message=Retry run out of 75 retry times, message=failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory)>, <Time:{'RPC start': '2024-07-30 20:12:45.088180', 'RPC error': '2024-07-30 20:16:16.003652'}>
RPC error: [insert_rows], <MilvusException: (code=<bound method _MultiThreadedRendezvous.code of <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-07-30T20:16:16.002968128+08:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory"}"
>>, message=Retry run out of 75 retry times, message=failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory)>, <Time:{'RPC start': '2024-07-30 20:12:45.088144', 'RPC error': '2024-07-30 20:16:16.003949'}>
Traceback (most recent call last):
  File "/opt/data/private/shuowang/./image_search_wulj.py", line 33, in <module>
    client.insert(
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/milvus_client/milvus_client.py", line 223, in insert
    raise ex from ex
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/milvus_client/milvus_client.py", line 219, in insert
    res = conn.insert_rows(
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/decorators.py", line 147, in handler
    raise e from e
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/decorators.py", line 143, in handler
    return func(*args, **kwargs)
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/decorators.py", line 182, in handler
    return func(self, *args, **kwargs)
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/decorators.py", line 122, in handler
    raise e from e
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/decorators.py", line 87, in handler
    return func(*args, **kwargs)
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 492, in insert_rows
    request = self._prepare_row_insert_request(
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 513, in _prepare_row_insert_request
    schema = self.describe_collection(collection_name, timeout=timeout)
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/decorators.py", line 147, in handler
    raise e from e
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/decorators.py", line 143, in handler
    return func(*args, **kwargs)
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/decorators.py", line 182, in handler
    return func(self, *args, **kwargs)
  File "/opt/conda/envs/image_search/lib/python3.10/site-packages/pymilvus/decorators.py", line 93, in handler
    raise MilvusException(e.code, f"{to_msg}, message={e.details()}") from e
pymilvus.exceptions.MilvusException: <MilvusException: (code=<bound method _MultiThreadedRendezvous.code of <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-07-30T20:16:16.002968128+08:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory"}"
>>, message=Retry run out of 75 retry times, message=failed to connect to all addresses; last error: UNKNOWN: unix:/tmp/tmp_i3bafr0_image_search_wulj_test.db.sock: No such file or directory)>

Could you give me some advice for this error?

Anything else?

No response

XuanYang-cn commented 1 month ago

assign @junjiejiangjjj

XuanYang-cn commented 1 month ago

/assign @junjiejiangjjj

sre-ci-robot commented 1 month ago

@XuanYang-cn: GitHub didn't allow me to assign the following users: junjiejiangjjj.

Note that only milvus-io members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide

In response to [this](https://github.com/milvus-io/pymilvus/issues/2205#issuecomment-2262826127): >/assign @junjiejiangjjj Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

junjiejiangjjj commented 1 month ago

Hi @wulongjian , do you have the logs before retry 1? Or start a gRPC service in this way, use client = MilvusClient("http://127.0.0.1:9999") to connect server，This way, the error logs will be clearer.

>>> from milvus_lite.server import Server
>>> s = Server('image_search_wulj_test.db', '127.0.0.1:9999')
>>> s.start()
True
>>> s._p.wait()

milvus-io / pymilvus