search

zilliztech / milvus-backup

Backup and restore tool for Milvus
Apache License 2.0
129 stars 46 forks source link

[Bug]: run milvus-backup in k8s pod, retrurn "failed to connect \"Addr\": \"milvus:tcp:\" " #197

Open jackiedlh opened 1 year ago

jackiedlh commented 1 year ago

Current Behavior

When I run milvus-backup command on k8s pod, the target milvus address is wrong.

in backup.yaml:

milvus: address: milvus port: 19530 authorizationEnabled: false

in log:

[2023/09/08 07:35:33.943 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=] [2023/09/08 07:35:33.943 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log] [2023/09/08 07:35:33.943 +00:00] [INFO] [core/backup_impl_create_backup.go:25] ["receive CreateBackupRequest"] [requestId=4b980b8b-4e1a-11ee-8398-82b39bdfe663] [backupName=backup] [collections="[]"] [databaseCollections=] [async=false] [2023/09/08 07:35:34.168 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {\n \"Addr\": \"milvus:tcp:\",\n \"ServerName\": \"milvus:tcp:\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp: address milvus:tcp:: too many colons in address\""]

Expected Behavior

expect connect to:

milvus:19530

Steps To Reproduce

1 copy milvus-backup tool to k8s milvus proxy pod
2 copy config files to milvus pod
3 run command:  ./milvus-backup create -n backup

Environment

aws EKS

Anything else?

No response

wayblink commented 1 year ago

address should be a ip or a domain.

Archalbc commented 1 year ago

address should be a ip or a domain.

"milvus" is correct in a Kubernetes context, it will resolve to the milvus-proxy pod, actually this look like a bug.

In debug mode you can have the proof that "milvus" is correctly resolved but badly interpreted by sub-libraries : 

[2023/09/18 10:24:01.536 +00:00] [DEBUG] [core/backup_context.go:58] ["Start Milvus client"] [endpoint=milvus:tcp://10.224.44.247:19530]
[2023/09/18 10:24:01.536 +00:00] [WARN] [grpclog/grpclog.go:46] ["[core][Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {\n  \"Addr\": \"milvus:tcp:\",\n  \"ServerName\": \"milvus:tcp:\",\n  \"Attributes\": null,\n  \"BalancerAttributes\": null,\n  \"Type\": 0,\n  \"Metadata\": null\n}. Err: connection error: desc = \"transport: Error while dialing dial tcp: address milvus:tcp:: too many colons in address\""]

Same issue using : milvus.my-namespace.svc.cluster.local

I guess related code is : https://github.com/xushaoxiao/milvus-backup/blob/80c754eb9bfbb96961f244404ba7a3a5054d3e24/core/backup_context.go#L63C22-L63C50 and https://github.com/milvus-io/milvus-sdk-go/blob/master/client/client.go

The used func is deprecated as described in comment : https://github.com/milvus-io/milvus-sdk-go/blob/master/client/client.go#L285

No issue with v0.3.2

[2023/09/18 13:01:26.991 +00:00] [DEBUG] [core/backup_context.go:58] ["Start Milvus client"] [endpoint=milvus:443]
nightmorning0 commented 1 year ago

It is definitely a bug. Add PORT# at the tail of the address. Something likes this:

milvus:
address: milvus:19530#
port: 19530
authorizationEnabled: false

It will comment :tcp: from milvus:tcp:: in your error as a tricky solution.

adeelamin15 commented 8 months ago

Yup it is definitely a bug. For me workaround is to set the port using MILVUS_PORT environment variable. Apparently the bug is causing garbage value in port field which can be overridden using environment variable.

wayblink commented 4 months ago

/assign