search

amazon-archives / aws-sdk-unity

ARCHIVED: The aws sdk for unity is now distributed as a part of aws sdk for dotnet:
https://github.com/aws/aws-sdk-net
Other
105 stars 43 forks source link

Out of memory crash when uploading to S3 with Unity API on Android #85

Closed jrunt closed 8 years ago

jrunt commented 9 years ago

We are using Amazon S3 to store and retrieve files in our mobile application done using Unity3D. When sending files exceeding around 20MB and more, the application crashes to out of memory on Android devices.

We are using PostObjectAsync method and a FileStream as source as explained both in online documents (http://docs.aws.amazon.com/mobile/sdkforunity/developerguide/s3.html) and in S3Example.cs which came with AWSSDK. The particular part of code goes as following:

public void PostS3ObjectFromFile (string s3filename, string filepath) {
  Stream fileStream = new FileStream (filepath, FileMode.Open, FileAccess.Read, FileShare.Read);

  var request = new PostObjectRequest()
  {
    Bucket = s3BucketName,
    Key = s3filename,
    InputStream = fileStream,
    CannedACL = S3CannedACL.Private
  };

  S3Client.PostObjectAsync (request, (responseObj) => { // Crashes when file is >~20MB
    if (responseObj.Exception == null) {
      Debug.Log (string.Format ("object {0} posted to bucket {1}", responseObj.Request.Key, responseObj.Request.Bucket));
    } else {
      Debug.Log (string.Format ("\n receieved error {0}", responseObj.Response.HttpStatusCode.ToString ()));
    }
  });
}

According to Unity's profiler, Mono takes several hundreds of megabytes memory when the request is executed and then the application force quits. Shouldn't the memory stamp be small when dealing with direct streams? We have also tried to decrease and increase the buffer size in AWSSDKUtils.cs (DefaultBufferSize from 8192) without any success.

We haven't found any solution to the issue and don't quite know how to proceed.

karthiksaligrama commented 9 years ago

thanks for reporting this.

because the WWW doesn't directly accept streams we try to use copy over the data to the memory stream convert it to byte[] and use it to post the data to aws servers. So its expected to have higher memory usage than normal.

In any case i'll take a look at this. Do you happen to have a stack trace of the error? and can you also let us know the version of the sdk that you are using?

jrunt commented 9 years ago

Thank you for quick response!

We faced this up issue with SDK 2.0.0.2, but the same issue occurred also with 2.0.0.4 we updated this week.

No matter of the file size (after certain point) we try to upload, we always will get the "Out of memory on a 23393512-byte allocation." briefly before crash. Here's the stack trace in moment of crash:

I/Unity   (16147): (Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
I/Unity   (16147): 
I/Unity   (16147): Preparing the filestream
I/Unity   (16147):  
I/Unity   (16147): (Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
I/Unity   (16147): 
I/Unity   (16147): Posting S3 object
I/Unity   (16147):  
I/Unity   (16147): (Filename: ./artifacts/generated/common/runtime/UnityEngineDebug.gen.cpp Line: 56)
I/Unity   (16147): 
D/ProgressBar(24964): setProgress = 0
D/ProgressBar(24964): setProgress = 0, fromUser = false
D/ProgressBar(24964): mProgress = 0mIndeterminate = false, mMin = 0, mMax = 228087
D/STATUSBAR-NetworkController( 2748): refreshSignalCluster: data=-1 bt=false
D/STATUSBAR-IconMerger( 2748): checkOverflow(960), More:false, Req:false Child:8
D/ProgressBar(24964): setProgress = 0
D/ProgressBar(24964): setProgress = 0, fromUser = false
D/ProgressBar(24964): mProgress = 0mIndeterminate = false, mMin = 0, mMax = 228087
D/SensorService( 2379):   0.0 0.2 10.0
D/ProgressBar(24964): setProgress = 0
D/ProgressBar(24964): setProgress = 0, fromUser = false
D/ProgressBar(24964): mProgress = 0mIndeterminate = false, mMin = 0, mMax = 228087
D/dalvikvm(16147): GC_FOR_ALLOC freed 1120K, 19% free 10985K/13436K, paused 16ms, total 16ms
D/ProgressBar(24964): setProgress = 0
D/ProgressBar(24964): setProgress = 0, fromUser = false
D/ProgressBar(24964): mProgress = 0mIndeterminate = false, mMin = 0, mMax = 228087
I/dalvikvm-heap(16147): Grow heap (frag case) to 30.795MB for 19706966-byte allocation
D/dalvikvm(16147): GC_FOR_ALLOC freed 1K, 8% free 30229K/32684K, paused 21ms, total 21ms
V/MediaPlayer-JNI(16147): native_finalize
V/MediaPlayer-JNI(16147): release
D/dalvikvm(16147): GC_CONCURRENT freed 9K, 8% free 30225K/32684K, paused 33ms+2ms, total 53ms
D/ProgressBar(24964): setProgress = 0
D/ProgressBar(24964): setProgress = 0, fromUser = false
D/ProgressBar(24964): mProgress = 0mIndeterminate = false, mMin = 0, mMax = 228087
D/ProgressBar(24964): setProgress = 0
D/ProgressBar(24964): setProgress = 0, fromUser = false
D/ProgressBar(24964): mProgress = 0mIndeterminate = false, mMin = 0, mMax = 228087
D/SensorService( 2379):   -0.0 0.2 10.0
D/dalvikvm(16147): GC_FOR_ALLOC freed 1556K, 10% free 31685K/34824K, paused 22ms, total 22ms
I/dalvikvm-heap(16147): Grow heap (frag case) to 35.002MB for 2921704-byte allocation
D/dalvikvm(16147): GC_FOR_ALLOC freed 1425K, 13% free 33113K/37680K, paused 26ms, total 26ms
D/dalvikvm(16147): GC_CONCURRENT freed 2K, 13% free 33112K/37680K, paused 3ms+2ms, total 25ms
D/dalvikvm(16147): WAIT_FOR_CONCURRENT_GC blocked 22ms
I/dalvikvm-heap(16147): Grow heap (frag case) to 39.185MB for 5846248-byte allocation
D/ProgressBar(24964): setProgress = 0
D/ProgressBar(24964): setProgress = 0, fromUser = false
D/ProgressBar(24964): mProgress = 0mIndeterminate = false, mMin = 0, mMax = 228087
D/SSRMv2:Monitor( 2379): SIOP:: AP = 390, Prev AP = 390, Duration = 10012
D/dalvikvm(16147): GC_FOR_ALLOC freed 2853K, 18% free 35967K/43392K, paused 30ms, total 30ms
D/dalvikvm(16147): GC_CONCURRENT freed <1K, 18% free 35968K/43392K, paused 2ms+2ms, total 22ms
D/dalvikvm(16147): WAIT_FOR_CONCURRENT_GC blocked 19ms
I/dalvikvm-heap(16147): Grow heap (frag case) to 47.552MB for 11695336-byte allocation
D/STATUSBAR-NetworkController( 2748): onReceive() - RSSI_CHANGED_ACTION, WIFI_STATE, NETWORK_STATE
D/STATUSBAR-NetworkController( 2748): refreshSignalCluster: data=-1 bt=false
D/STATUSBAR-IconMerger( 2748): checkOverflow(960), More:false, Req:false Child:8
D/dalvikvm(16147): GC_CONCURRENT freed 5709K, 24% free 41679K/54816K, paused 17ms+3ms, total 38ms
D/dalvikvm(16147): WAIT_FOR_CONCURRENT_GC blocked 21ms
D/dalvikvm(16147): WAIT_FOR_CONCURRENT_GC blocked 21ms
D/dalvikvm(16147): WAIT_FOR_CONCURRENT_GC blocked 7ms
I/dalvikvm-heap(16147): Forcing collection of SoftReferences for 23393512-byte allocation
D/dalvikvm(16147): GC_BEFORE_OOM freed 71K, 25% free 41609K/54816K, paused 28ms, total 28ms
E/dalvikvm-heap(16147): Out of memory on a 23393512-byte allocation.
I/dalvikvm(16147): "Thread-6231" prio=5 tid=18 RUNNABLE
I/dalvikvm(16147):   | group="main" sCount=0 dsCount=0 obj=0x426c0f90 self=0x59d94460
I/dalvikvm(16147):   | sysTid=16399 nice=0 sched=0/0 cgrp=apps handle=1559294696
I/dalvikvm(16147):   | state=R schedstat=( 181569293 24035918 243 ) utm=11 stm=6 core=1
I/dalvikvm(16147):   at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:~91)
I/dalvikvm(16147):   at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:201)
I/dalvikvm(16147):   at libcore.net.http.RetryableOutputStream.write(RetryableOutputStream.java:61)
I/dalvikvm(16147):   at com.unity3d.player.WWW.run((null):-1)
I/dalvikvm(16147): 
W/dalvikvm(16147): threadid=18: thread exiting with uncaught exception (group=0x41adf700)
W/System.err(16147): java.lang.OutOfMemoryError
W/System.err(16147):    at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:91)
W/System.err(16147):    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:201)
W/System.err(16147):    at libcore.net.http.RetryableOutputStream.write(RetryableOutputStream.java:61)
W/System.err(16147):    at com.unity3d.player.WWW.run(Unknown Source)
W/FlurryAgent(16147): Error logged: uncaught
D/LocationManagerService( 2379): getProviders()=[]
D/LocationManagerService( 2379): getProviders()=[]
D/LocationManagerService( 2379): getBestProvider(Criteria[power=NO_REQ acc=---], true)=null
W/FlurryAgent(16147): End session with context: com.neatplug.u3d.plugins.common.NPUnityPlayerActivity@425bd470 count:0
W/FlurryAgent(16147): Finalize session
E/AndroidRuntime(16147): FATAL EXCEPTION: Thread-6231
E/AndroidRuntime(16147): java.lang.Error: FATAL EXCEPTION [Thread-6231]
E/AndroidRuntime(16147): Unity version     : 5.0.2p4
E/AndroidRuntime(16147): Device model      : samsung GT-I9305
E/AndroidRuntime(16147): Device fingerprint: samsung/m3xx/m3:4.3/JSS15J/I9305XXUEMKC:user/release-keys
E/AndroidRuntime(16147): 
E/AndroidRuntime(16147): Caused by: java.lang.OutOfMemoryError
E/AndroidRuntime(16147):    at java.io.ByteArrayOutputStream.expand(ByteArrayOutputStream.java:91)
E/AndroidRuntime(16147):    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:201)
E/AndroidRuntime(16147):    at libcore.net.http.RetryableOutputStream.write(RetryableOutputStream.java:61)
E/AndroidRuntime(16147):    at com.unity3d.player.WWW.run(Unknown Source)
I/ActivityManager( 2379): Notify an ApplicationCrash
karthiksaligrama commented 9 years ago

thanks i'll take a look at it.. another question though what version of unity are you using?

jrunt commented 9 years ago

We have used Unity version 5.0.2p4 and we have also tried with 5.1.0p1 with same results. Would our best option be to implement own uploading using web-access or is it likely to have some fix related to this issue in the Unity SDK itself?

karthiksaligrama commented 8 years ago

PostObject doesn't actually convert the input stream to memory stream it uses the stream directly based on your stack trace it looks like this is a bug in the WWW android implementation. I would raise a bug report with unity.