search

roeap / object-store-python

Python bindings and arrow integration for the rust object_store crate.
Apache License 2.0
51 stars 8 forks source link

Add support for put_multipart #13

Open rupurt opened 5 months ago

rupurt commented 5 months ago

Howdy,

Love the bindings so far! Are there any plans to add support for put_multipart?

Cheers

rupurt commented 5 months ago

I've started a PR to try and add these functions in https://github.com/roeap/object-store-python/pull/14. Do you have any suggestions or docs on how I should be adding these?

arogozhnikov commented 1 month ago

Is there a limitation on uploaded object size given that multipart upload is not part of API?

roeap commented 1 month ago

there should not be other then the upstream store. of course very large objects can always fail since there is more room for network errors during a large time period ..

That said, multi-part uploads mainly allow for parallelizing the upload and should have a significant performance impact on large amounts of data - especially for single very large files.

arogozhnikov commented 1 month ago

there should not be other then the upstream store

thx, good to know (just thinking out loud: then for S3 that's 5GB max)