Build multipart upload

[yuyashiraki]

Summary:

Context

We found that AWS-SDK S3 API would fail when we try to write more than 5GB of data. It is a blocking us to do capacity testing for a larger FARGATE container.

As mentioned in the post, one of our options is to use multi-part upload developed by AWS to split a file into smaller chunks to form the single S3 file. We have discussed with AWS engineers and decided to develop multipart upload logic.

Technical Details

We are splitting a file into 300MB (314572800 bytes) ByteStreams. Then, using multipart upload to upload each part to construct the single S3 file.

The multipart upload consists of three steps.

1. creating the multipart upload instance via create_multipart_upload(). it would generate upload id.
2. sending bytestreams to S3 using upload_part()
3. completing multipart upload via complete_multipart_upload() after uploading all the parts

Ref

Below are the references we used to develop multipart functionality.

• https://github.com/awslabs/aws-sdk-rust/pull/519/files
• https://github.com/awslabs/smithy-rs/pull/1361/files
• https://docs.rs/aws-smithy-http/0.48.0/aws_smithy_http/byte_stream/struct.ByteStream.html#method.read_from
• https://docs.rs/aws-smithy-http/0.48.0/aws_smithy_http/byte_stream/struct.FsBuilder.html#method.offset
• https://docs.rs/aws-sdk-s3/0.18.0/aws_sdk_s3/client/struct.Client.html#method.create_multipart_upload

Differential Revision: D39534523

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D39534523