Add retries and chunking parameters to upload_folder

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution

https://clear.ml/docs

Apache License 2.0

5.66k stars 651 forks source link

Add retries and chunking parameters to upload_folder #1305

Open jennzhugewps opened 3 months ago

jennzhugewps commented 3 months ago

Proposal Summary

Hi ClearML team, it would be nice if upload_folder had the retries parameter. It would also be nice to have a chunking parameter for upload_folder and upload_file as well.

Motivation

I need to upload data over spotty wifi connections.

Related Discussion

https://clearml.slack.com/archives/CTK20V944/p1722022515201179

eugen-ajechiloae-clearml commented 2 months ago

Hi @jennzhugewps ! We have prepared a fix for this issue that we will release soon

pollfly commented 2 months ago

Hey @jennzhugewps! v1.16.3 is now out with a retries parameter for StorageManager.upload_folder() .