Use Common Runtime Libraries in S3TransferManager - Githubissues

wellcomecollection / storage-service

🗃️ Managing the safe, long-term storage of our digital collections in the cloud

MIT License

31 stars 5 forks source link

Use Common Runtime Libraries in S3TransferManager #1122

Closed kenoir closed 3 months ago

kenoir commented 3 months ago

What does this change?

This change attempts to make use of the Common Runtime Libraries for the AWS SDK intended to improve performance.

We include them as a dependency but do not use them in the S3TransferManager where we do large transfers. The documentation describes how to do this.

The intention is to improve the performance of the replicators which are currently showing issues transferring large files to S3.

How to test

[x] Do the tests pass?
[ ] When deployed do bags with large files get replicated succesfully?

How can we measure success?

Improved performance, less failures.

Have we considered potential risks?

This will impact replication performance, hopefully only positively but there are could be negative impacts.

kenoir commented 3 months ago

Closing as this doesn't change the implementation (the default behaviour is this hidden in the s3 transfer manager builder.