[C++] configurable multipart upload size

apache / arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

https://arrow.apache.org/

Apache License 2.0

14.67k stars 3.56k forks source link

[C++] configurable multipart upload size #44251

Open shaoting-huang opened 2 months ago

shaoting-huang commented 2 months ago

Describe the enhancement requested

Currently the multi part upload size is set to 10MB because of Cloudfare R2. That means write memory should be buffered at least 10MB to be uploaded to cloud storage. However, there are different minimum requirement for clouds. (AWS 5MB, Aliyun 100KB, Tencent 1MB, Azure 512 byte). Hope the kPartUploadSize can be configurable to support more memory control.

Component(s)

C++

mapleFU commented 1 month ago

Would you mind contribute this?