search

aws-amplify / aws-sdk-ios

AWS SDK for iOS. For more information, see our web site:
https://aws-amplify.github.io/docs
Other
1.68k stars 885 forks source link

Extra disk space usage and delayed uploads when uploading large files to S3 #5439

Closed zamzamfp closed 1 month ago

zamzamfp commented 1 month ago

State your question We are facing an issue with the transferUtility when using multipart upload to upload large files (e.g., 10GB). The SDK chunks the entire file into smaller parts at the beginning, as stated in this question also, before the upload even starts. This behaviour requires the device to have enough additional storage space (e.g., 10GB extra) to accommodate the chunked file. It also significantly delays the upload process, as it can take up to a minute for the chunking to complete, with no clear indication of this happening unless debug logging is enabled.

Questions:

  1. Is there a technical reason the chunking process has to occur entirely before the upload starts, rather than chunking parts on-the-fly (i.e., chunk the first parts, upload them, and then proceed with the rest)?
  2. Are there any plans to change this behavior in the future to improve efficiency?
  3. Is there a way to track the progress of the chunking process so we can communicate it in the UI?

Which AWS Services are you utilizing? S3

Environment(please complete the following information):

phantumcode commented 1 month ago

@zamzamfp Thanks for submitting the issue. We will investigate and provide updates here.

zamzamfp commented 1 month ago

Hi @phantumcode, do you have any updates?

ruisebas commented 1 month ago

Hi @zamzamfp, answering your questions:

  1. Multipart upload has been implemented a long time ago and this was the approach that was chosen at the time.
  2. We currently have no plans on refactoring this implementation
  3. There's no way to track the chunking process.

The Amplify Library for Swift behaves the way you suggested and only creates the partial temp files right before they are uploaded.

Unless your application absolutely requires the usage of the AWS SDK for iOS for some use case, we recommend you consider migrating to Amplify.

zamzamfp commented 1 month ago

Hi @ruisebas,Thanks for the response. Would you say the Amplify Library for Swift provides a better solution for uploading large files to S3, especially when compared to transferUtility.uploadUsingMultiPart, including support for background uploads?

ruisebas commented 1 month ago

Yes, the AWS Amplify Library for Swift is a newer multiplatform library that supports many common mobile use cases and takes advantage of Swift's latests improvements and features (e.g. async/await). It's built on top of the new AWS SDK for Swift.

Regarding uploading large files specifically, it does support background uploads (within the limitations imposed by the running Operation System). You can learn more about Amplify Storage in our documentation page.

ruisebas commented 1 month ago

@zamzamfp since you've also raised https://github.com/aws-amplify/aws-sdk-ios/issues/5440, I should call out that unlike the Transfer Utility, Amplify Storage currently does not support resuming interrupted uploads due to the app being terminated.

ruisebas commented 1 month ago

Closing as all questions have been answered.

github-actions[bot] commented 1 month ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.