search

aws-solutions / data-transfer-hub

Seamless User Interface for replicating data into AWS.
https://aws-solutions.github.io/data-transfer-hub/en/
Apache License 2.0
136 stars 22 forks source link

Optimize the transfer of large files ( object size > 100GB) #116

Closed YikaiHu closed 1 year ago

YikaiHu commented 1 year ago

Describe the bug When the individual file size exceeds 100GB, DTH is prone to transmission failures.

Moreover, this file will only be transferred by one EC2 Jobworker in the cluster, resulting in low efficiency.

To Reproduce Transfer a S3 object which size is larger than 100GB.

Please complete the following information about the solution:

YikaiHu commented 1 year ago

Completed in v2.5.0