storage: Support parallel composite uploads for faster transfers

[stanhu]

Is your feature request related to a problem? Please describe.

gsutil supports parallel composite uploads, but the Google Cloud Storage SDKs do not.

For large files, transfers to Google Cloud Storage could be significantly faster if chunks of data are uploaded simultaneously. The Amazon SDKs natively support multi-threaded, multipart uploads with Golang, Ruby, and Python libraries.

Describe the solution you'd like

gsutil implements this in Python.

Ideally there would be a simple boolean parameter, such as ParallelCompositeUpload, that would activate this mode in the bucket Write.

Describe alternatives you've considered

This could be implemented in the application layer, but it really belongs in the SDK.

Additional context Add any other context or screenshots about the feature request here.

[tritone]

I'd say this is comparable to #1672. It's a very reasonable feature request, but would entail more involved client-side logic than we usually include in this library, which is a fairly thin client. I'll raise internally whether this can be prioritized.

[quartzmo]

@tritone Can you please ask internally about the similar request to add this in google-cloud-ruby as well?

googleapis/google-cloud-ruby#8147

[quartzmo]

@tritone Any update as to whether or not this feature can be prioritized?

[tritone]

@quartzmo I don't think we are prioritizing this in the next quarter at least. @frankyn may have more context for Ruby specifically.

[frankyn]

We are not yet prioritizing this work across languages. It's still in our backlog, and we are working our way back to it.

[Slach]

@frankyn why you don't want implement it?

[frankyn]

We do, but it's competing with other work items don't have ETA yet. Thanks for your patience.

[tritone]

Just wanted to provide a long-delayed update here. We're planning on implementing parallel upload and download features in the newly-created subpackage transfermanager. We did the initial merge for downloads this week, more download features are coming later this month, and uploads should be coming in the second half of the year most likely. Feel free to take a look here: https://pkg.go.dev/cloud.google.com/go/storage@v1.41.1-0.20240605002059-de9872df8565/transfermanager ; we are open to hearing feedback.

[Slach]

@tritone great news, hope you will update this issue when Uploader will implements

[Vytenis-Valutkevicius]

Hi, @tritone, I am rather interested in this feature, so since it's been few months since last update just wanted to as, maybe you know if it is currently scheduled or some ETA is assigned? Thanks.