search

minio / mc

Unix like utilities for object store
https://min.io/download
GNU Affero General Public License v3.0
2.86k stars 548 forks source link

`mc cp` retry copying when src(or dst) reset its connection #4959

Closed hikarika closed 4 months ago

hikarika commented 4 months ago

Expected behavior

The mc cp command should retry once or more at default when source or destination's connection is reseted by peer.

Cloud service providers may limit bandwidth or TPS by reseting excessive connections, but temporarily usually. Maybe it's a good idea to retry once or more when mc encounter this kind of problems to tolerate this.

Actual behavior


mc: <ERROR> Failed to copy `https://s3.cloudA.com//${bucket1_name}/${DIR}/${object_file}`. read tcp ${IP1}:${PORT1}->${IP2}:443: read: connection reset by peer
mc: <ERROR> Failed to copy `https://s3.cloudA.com/${bucket1_name}/${DIR}/${object_file}`. Put "https://s3.cloudB.com/${bucket2_name}/${DIR}/${object_file}?partNumber=XXX&uploadId=XXX": net/http: HTTP/1.x transport connection broken: http: ContentLength=16777216 with Body length 8204959

executed cmd： mc cp -r cloudA/${bucket1_name}/${DIR}/ cloudB/${bucket2_name}/${DIR}/

Steps to reproduce the behavior

In our case, I copied nearly 80,000 files from CloudA to CloudB using mc cp, and then CloudA limits my access by reset the connection, causing an exception when shown above that dest CloudB found no more data to upload.

mc --version

mc version RELEASE.2024-03-30T15-29-52Z (commit-id=9f8147bf0e037730077a1b3baef25e53181099b0) Runtime: go1.21.8 linux/amd64 Copyright (c) 2015-2024 MinIO, Inc. License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

System information

CentOS 8 - amd64

hikarika commented 4 months ago

duplicate with #4958