zoewangg
commented
1 year ago Apologies for the delayed response.
If an InputStream is provided then multi-part upload becomes sequential where all parts are lined up in sequence and uploads are performed.
Reading from the input stream is sequential but uploading could be concurrent. Under the hood, AWS CRT-based S3 client buffers the part and whenever a part is available, it will send it, so it is possible that multipart parts are being uploaded at the same time.
This can be achieved by assigning an InputStream to a PutObjectRequest request in the multi-part upload process and carrying out the transfer in parallel.
This is an interesting idea, but how would the SDK know how many parts are available. I guess the SDK could take a list of inputStream and infer the number of the size of the list. Alternatively, we may be able to provide better support for seekable AsyncRequestBody, say SeekableAsyncRequestBody, that allows the SDK to read data from any offset just like reading from a file. This way, we may be able to achieve high throughput.
vikasvb90
denizk
Describe the feature
Currently, transfermanager supports multi-part parallel upload only using file as an input. If an InputStream is provided then multi-part upload becomes sequential where all parts are lined up in sequence and uploads are performed. This is a proposal to provide support to accept multiple InputStreams each emitted for a specific portion of content and to perform concurrent uploads where each stream is responsible for the transfer of a specific part. At the end of the upload, complete upload request is triggered to merge the uploaded parts.
Use Case
There are scenarios where pre-processing of content may be required before its transfer. Let's say that this pre-processing work can be carried out independently for different parts of the content. Now, to carry out both pre-processing and transfer of content, we have two options available. Either content is read, pre-processed and transferred serially using InputStream or content is first pre-processed completely by pre-processing different parts concurrently and then making the transfer as a separate operation. In first approach, we avoid duplicate reads but end up making the whole process latent and in second approach we lose streaming nature of the content and end up reading the content twice.
Proposed Solution
What we need here is the ability to apply the package of all streaming operations on a unit of content vs applying individual operation on the whole content. Multiple InputStreams emitted for different parts of content can solve this problem where each stream applies all streaming operations like pre-processing, transfer, post-processing on a unit of content. This can be achieved by assigning an InputStream to a
PutObjectRequestrequest in the multi-part upload process and carrying out the transfer in parallel.Other Information
We were able to achieve multi-part parallel upload using streams for async uploads using S3AsyncClient. We were also able to customize v1 transfermanager and achieve the same there as well. We haven't got a chance to accommodate these changes in v2 transfermanager though. Following is a reference PR of async upload : https://github.com/opensearch-project/OpenSearch/pull/7217/files#diff-a472eda8bf5f051224172440b745603502d3545f8326b28d6014a79d5a833cd5
An iterable pattern is used in the class to create a stream for each part.
Stream stream = streamContext.getStreamProvider().provideStream(partIdx)Acknowledgements
AWS Java SDK version used
2.20.26
JDK version used
11
Operating System and version
OSx