search

googleapis / google-cloud-ruby

Google Cloud Client Library for Ruby
https://googleapis.github.io/google-cloud-ruby/
Apache License 2.0
1.34k stars 536 forks source link

storage: Support parallel composite uploads for faster transfers #8147

Open stanhu opened 3 years ago

stanhu commented 3 years ago

Is your feature request related to a problem? Please describe.

gsutil supports parallel composite uploads, but the Google Cloud Storage SDKs do not.

For large files, transfers to Google Cloud Storage could be significantly faster if chunks of data are uploaded simultaneously. The Amazon SDKs natively support multi-threaded, multipart uploads with Golang, Ruby, and Python libraries.

Describe the solution you'd like

gsutil implements this in Python.

Ideally there would be a simple boolean parameter, such as parallel_composite_upload:

# Copy the file to a backup bucket
backup = storage.bucket "task-attachment-backups"
file.copy backup, file.name, parallel_composite_upload: true

Describe alternatives you've considered

This could be implemented in the application layer, but it really belongs in the SDK.

Additional context

I created an equivalent issue in the Go SDK: https://github.com/googleapis/google-cloud-go/issues/3219

quartzmo commented 3 years ago

See @tritone's comment on the similar google-cloud-go feature request:

It's a very reasonable feature request, but would entail more involved client-side logic than we usually include in this library, which is a fairly thin client. I'll raise internally whether this can be prioritized.

quartzmo commented 3 years ago

New comment from @frankyn:

We are not yet prioritizing this work across languages. It's still in our backlog, and we are working our way back to it.