search

seung-lab / cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.
BSD 3-Clause "New" or "Revised" License
38 stars 8 forks source link

Multipart Uploads / Composite Uploads #7

Closed william-silversmith closed 6 months ago

william-silversmith commented 4 years ago

It'd be nice if CF could transparently take advantage of multipart uploads for very large files.

https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html https://cloud.google.com/storage/docs/composite-objects

william-silversmith commented 4 years ago

For google, this capability seems to be you upload smaller individual files then compose them into a single unit. That could be pretty useful for multi-process upload, less so for multi-threaded upload.

william-silversmith commented 6 months ago

Resolved in https://github.com/seung-lab/cloud-files/pull/86 and https://github.com/seung-lab/cloud-files/pull/85