search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.69k stars 655 forks source link

Specify multipart threshold and chunksize in boto3 configuration #1058

Closed cgaudreau-ubisoft closed 1 year ago

cgaudreau-ubisoft commented 1 year ago

Proposal Summary

In the boto3 configuration section of the clearml config it should be possible to specify the threshold and chunksize for doing multipart uploads using S3.

Motivation

We would like to bypass the multipart upload in certain cases, which means we need to be able to manually set the threshold for the multipart upload.

Related Discussion

N/A

ainoam commented 1 year ago

Thanks for suggesting @cgaudreau-ubisoft

We'll look into adding support for this.