search

aws-amplify / amplify-flutter

A declarative library with an easy-to-use interface for building Flutter applications on AWS.
https://docs.amplify.aws
Apache License 2.0
1.31k stars 243 forks source link

AWSStreamedHttpRequest consumes a lot of memory on iOS when uploading big file #4082

Closed sevoster closed 9 months ago

sevoster commented 10 months ago

Description

I'm trying to upload file from my application to Amazon S3 using this example: https://github.com/aws-amplify/amplify-flutter/blob/next/packages/aws_signature_v4/example/bin/example.dart

On iOS when I upload big video file (~250 Mb) this request consumes all the available memory and I get exception:

Exhausted heap space, trying to allocate 2147483664 bytes.
[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: Out of Memory
#0      _HttpOutgoing.addStream.<anonymous closure> (dart:_http/http_impl.dart)
#1      _FutureListener.handleError (dart:async/future_impl.dart)
#2      Future._propagateToListeners.handleError (dart:async/future_impl.dart)
#3      Future._propagateToListeners (dart:async/future_impl.dart)
#4      Future._completeError (dart:async/future_impl.dart)
#5      Future._asyncCompleteError.<anonymous closure> (dart:async/future_impl.dart)
#6      _microtaskLoop (dart:async/schedule_microtask.dart)
#7      _startMicrotaskLoop (dart:async/schedule_microtask.dart)
#8      _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart)
#9      _BufferingStreamSubscription._add (dart:async/stream_impl.dart)
#10     _StreamController._add (dart:async/stream_controller.dart)
#11     _StreamController.add (dart:async/stream_controller.dart)
#12     _RootZone.runUnaryGuarded (dart:async/zone.dart)
#13     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart)
#14     _BufferingStreamSubscription._add (dart:async/stream_impl.dart)
#15     _StreamController._add (dart:async/stream_controller.dart)
#16     _StreamController.add (dart:async/stream_controller.dart)
#17     Tap.tap.<anonymous closure> (package:stream_transform/src/tap.dart)
#18     TransformByHandlers.transformByHandlers.<anonymous closure>.<anonymous closure> (package:stream_transform/src/from_handlers.dart)
#19     _RootZone.runUnaryGuarded (dart:async/zone.dart)
#20     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart)
#21     _BufferingStreamSubscription._add (dart:async/stream_impl.dart)
#22     _StreamController._add (dart:async/stream_controller.dart)
#23     _StreamController.add (dart:async/stream_controller.dart)
#24     _AsyncStarStreamController.add (dart:async-patch/async_patch.dart)
#25     S3ServiceConfiguration.transformBody (package:aws_signature_v4/src/configuration/services/s3.dart)
#26     _FutureListener.handleValue (dart:async/future_impl.dart)
#27     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart)
#28     Future._propagateToListeners (dart:async/future_impl.dart)
#29     Future._completeWithValue (dart:async/future_impl.dart)
#30     _FutureListener.handleValue (dart:async/future_impl.dart)
#31     Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart)

Maybe you have some advise how to decrease memory consumption here.

Categories

Steps to Reproduce

No response

Screenshots

No response

Platforms

Flutter Version

3.13.9

Amplify Flutter Version

0.6.1

Deployment Method

Custom Pipeline

Schema

No response

Jordan-Nelson commented 10 months ago

Hello @sevoster - thanks for opening the issue. The crash shows around 2gb of memory used. This would not be expected for a file size of 250mb. We will take a look and see if we can reproduce this.

sevoster commented 9 months ago

Also I have a question about AWSStreamedHttpRequest. Does it support multipart file uploading? For example, if I want to upload big files (1-2 GB) should I use it? Or I need to implement multipart upload myself since amplify doesn't support use cases with temporary credentials.

khatruong2009 commented 9 months ago

Hi @sevoster, I've tried uploading files that are around 250mb and the most memory it has used in the iOS simulator has been 800-900mb and then it goes back down to 150-300mb as soon as the upload is finished. I also created another sample flutter app to upload a 250mb file and got the same results as before. Could you give me reproduction steps to help me reproduce this issue?

sevoster commented 9 months ago

Hi @khatruong2009! I've tried to reproduce it again but it works fine for files ~300Mb now. I'm not sure what changed but seems like it's not relevant anymore. Feel free to close the issue. It would be great if you could also help me with uploading huge files to S3 (like 1-2Gb). In that case app crashes with memory limit and I suppose it is an expected behavior for AWSStreamedHttpRequest. I know that Storage plugin can perform multipart uploading for big files. Is it possible to somehow initialize Amplify with temporary credentials (AWS STS) in order to use Storage plugin?

khatruong2009 commented 9 months ago

Hi @sevoster, Using STS credentials with Amplify.configure() is not something that's currently supported because exposing SecretKey on the client side would be a security issue. A similar question was asked here (credit for the answer). I will be closing the issue now, please let us know if you run into any other issues in the future, thanks.