Retry upload/download Minio operations

CERT-Polska / karton

Distributed malware processing framework based on Python, Redis and S3.

https://karton-core.readthedocs.io/en/latest/

BSD 3-Clause "New" or "Revised" License

392 stars 45 forks source link

Closed psrok1 closed 2 years ago

psrok1 commented 2 years ago

Minio sets retry policy, but it handles only completed responses (if I understand correctly: connect/read problems are not retried by default - https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html#urllib3.util.Retry)

I think it would be safe to retry whole operation in case of these errors instead of changing only urllib3 retry policy.

closes #18, closes #170

psrok1 commented 2 years ago

I see that Boto3/Botocore is much better configured by default and already handles these timeouts and retries correctly.