minio / mc

Unix like utilities for object store
https://min.io/download
GNU Affero General Public License v3.0
2.86k stars 548 forks source link

Created a retry mechanism and added a --retry flag to mc mirror #4715

Closed zveinn closed 1 year ago

zveinn commented 1 year ago

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers under the terms of the Apache 2 license. By creating this pull request I represent that I have the right to license the contributions to the project maintainers under the Apache 2 license.

Description

There are failure cases when doing transfers that are not recoverable in the minio-go library. These cases can be recovered from at a higher level (more specifically in the mc client). This pull request implements a very basic retry mechanism which can be used to encapsulate any logic.

Notes

The retry mechanism is only encapsulating the mc mirror command for now. But we could expand it to encapsulate all transfers, eventually.

There are a few github issues which have been reported with failure cases (not bugs) which might be avoided with this retry mechanism: https://github.com/minio/mc/issues/4635 https://github.com/minio/mc/issues/4231 https://github.com/minio/mc/issues/3075 https://github.com/minio/mc/issues/3965 (this one might have been a filesystem issue) https://github.com/minio/mc/issues/4138 https://github.com/minio/mc/issues/4202

How to test this PR?

$ ./functional-tests.sh

Types of changes