search

abashev / vfs-s3

Amazon S3 driver for Apache commons-vfs (Virtual File System) project
Apache License 2.0
93 stars 50 forks source link

Improve reliability of large file uploads #25

Closed svella closed 9 years ago

svella commented 9 years ago
  1. Only use TransferManager for files larger than the multi-part threshold (16MB).
  2. Create and destroy TransferManager for each uploaded large file since relative overhead is very small and and some usage patterns are causing the thread pool to get leaked.
  3. Always upload from File object since that is the only case where the upload will be retried automatically by the AWS SDK.
  4. Allow configuration of maximum number of upload parallel threads since the default of 10 performs very poorly and seems to cause lots of errors, and can saturate network upload bandwidth.
svella commented 9 years ago

Retracting due to merge issues

abashev commented 9 years ago

svella commented 9 years ago

Merge issue is that it won't compile without dependency on newer version of AWS SDK which I didn't include because pending pull request #24, which I mistakenly thought had been accepted. Not sure what the best way to proceed would be.

abashev commented 9 years ago

I'll update it today.

svella commented 9 years ago

Now that SDK version is updated, this should be good to go.