Support/fix for transfers of more than 1TB size objects

[elderjoe]

Describe the bug As stated in the title, objects more than 1TB are having an issue with the transfer.

Using Data Transfer Hub and this plugin is the reason why this issue is created.

Use Case AMIs needs to be transferred to AWS China and these AMIs are more than 1TB.

Once these AMIs are transferred, EC2 instances will be created.

To Reproduce

• Create the task.
• Set the configuration of source and destination buckets.
• Scheduling does not matter, can be 1 time transfer or scheduled.

Expected behavior AMI should be transferred to the destination S3 bucket.

Please complete the following information about the solution:

• (SO8002) - Data Transfer Hub - S3 Plugin - Template version **v2.4.0

• [ ] Region: [ap-southeast-1, cn-north-1]

• [ ] Was the solution modified from the version published on this repository? NO

• [ ] Have you checked your service quotas for the services this solution uses? YES

• [ ] Were there any errors in the CloudWatch Logs? YES

Screenshots Error encountered:

Note: Role policy have no issue. S3 objects less than 1TB are successfully transferred. See Additional context.

Additional context AMIs less than 1TB are successfully transferred with the same steps in the To Reproduce.

Created cases in AWS however, it looks like no one is aware or have knowledge about the solution therefore resolving the issue may take a while.

[elderjoe]

Closing, this repository is not maintained anymore, I guess.

[evalzy]

The repository is keeping update, for the large file we suggest user split into 50MB to get optimal performance. Team is working on the alternative solution to support larger file transfer.

[elderjoe]

Hi @evalzy ,

I won't take much of your time, as we are looking for a solution in our predicament ASAP;

So, is it safe to say for now that for 1TB or higher is not supported by this plugin?

Also, when you meant, "split into 50mb", do you mean split the ami-123abc.bin to 50mb? That would be a lot for 1TB or more.

Looking forward to your reply.

Thanks.

[evalzy]

HI @elderjoe

If a single file reaches the GB level, the transmission failure rate increases. The larger the file, the higher the error rate.

We have the tools provide by S3 can be used for splitting, will back to you soon.

[elderjoe]

Hi @evalzy ,

Thank you.

[evalzy]

HI @elderjoe

You can refer below 2 method to split the big file into small piece then use the DTH to perform transfer task.

Refer 1 : https://medium.com/swlh/automatic-s3-file-splitter-620d04b6e81c

Refer 2: https://stackoverflow.com/questions/57351436/how-to-split-the-big-s3-file-into-multiple-files-based-on-size

[elderjoe]

Hello @evalzy,

Thank you. We will check these and see how it goes.

Really appreciated your help.