search

aws-amplify / aws-sdk-ios

AWS SDK for iOS. For more information, see our web site:
https://aws-amplify.github.io/docs
Other
1.68k stars 884 forks source link

Best practice for pausing upload in AWSS3TransferManger and AWSS3TransferUtility on network loss? #769

Closed jeffjvick closed 5 years ago

jeffjvick commented 6 years ago

I start a large upload with TransferManager and part way through I turn on airplane mode to kill the wifi connection. This gives a "The Internet connection appears to be offline." error as expected since it does not appear that TransferManager handles network drops on its own like TransferUtility does.

To combat this I'm trying to catch the network loss myself and pause the uploads before TransferManager errors. I'm using the Reachability cocoapod https://github.com/ashleymills/Reachability.swift to detect network loss and it seems like it's a race to see if I can pause before TransferManager errors. Usually I lose.

I have not seen any official sdk documentation on how to handle pausing on a network loss so I wanted to see if someone could point to another source. Or possibly I should be handling this in a different manner?

Also, please don't tell me to use TransferUtility because TransferManger is "old". TransferUtility does not support multipart uploads so whenever the network is lost it just restarts from the beginning and my app will be uploading large (100+ mb) files on less than ideal networks and I can't force users to keep restarting. I don't understand why multi-part wasn't a priority for the new code base but that's the way it is so it appears I'm stuck with the old TransferManager.

Thanks

chrisscholly commented 6 years ago

Exactly same problem here...

chrisscholly commented 6 years ago

TransferUtility should support multi-parts, otherwise it is definitely useless.

mutablealligator commented 6 years ago

Hi @jeffjvick,

Thanks for reporting to us. We are taking it as a feature request to introduce multi-part upload in TransferUtility. We will update this thread when we release it. Thank you for the understanding.

billykahl commented 6 years ago

Hi @kvasukib, Any timeline on when we can expect TransferUtility to have multi-part and pausing uploads instead of restarting the upload on network loss? I am having the same issue as @jeffjvick

Thank you for your help!

mutablealligator commented 6 years ago

Hello @jeffjvick, @billykahl, @chrisscholly,

We have implemented Mulit-part for the uploads in TransferUtility in 2.6.13. Please try it and let us know. See https://docs.aws.amazon.com/AWSiOSSDK/latest/Classes/AWSS3TransferUtility.html#//api/name/uploadDataUsingMultiPart:key:contentType:expression:completionHandler:NS_SWIFT_NAME:

jeffjvick commented 6 years ago

@kvasukib Oh wow, so first, thanks, I seriously didn't think you guys were actually working on this. :)

Unfortunately though I literally (today) just finished pushing the feature complete version of my app for the client and had been forced to make do with Transfer Manager. Doing so required a myriad of band-aids and work-arounds to deal with its issues (this was just one of many).

I spoke with them and they are willing to at least do some tests with Transfer Utility. We've gotten the old Transfer Manager stable to the point where most users can complete their uploads but there could be room for improvement so it's worth investigating.

I will let you know how those tests go.

chrisscholly commented 6 years ago

@kvasukib Thank you very much! Will take some time asap to test this new awesome feature :)

swaminator commented 6 years ago

Hi @jeffjvick. I'm a PM at AWS Mobile. Appreciate your contributions to this project. I'd like to chat with you sometime to get your feedback on our updates to TransferUtility. You can reach me at https://twitter.com/TheSwaminator

mutablealligator commented 6 years ago

@chrisscholly Thank you for the response! Please try it and let us know any feedback you have.

stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

fantapop commented 6 years ago

we'll be implementing this very soon. Probably within this week. Will let you know how it goes.

On Sun, Apr 15, 2018 at 10:07 PM, stale[bot] notifications@github.com wrote:

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aws/aws-sdk-ios/issues/769#issuecomment-381481092, or mute the thread https://github.com/notifications/unsubscribe-auth/AGWbqKFromPoNGLTsIDGFyuRaBNsX_n_ks5tpCcSgaJpZM4QrVxT .

jeffjvick commented 6 years ago

@kvasukib, @swaminator I've finally started to test the new multi-part for transfer utility. First, please update your aws-sdk-ios-samples to included a multi-part example. This would be helpful for others using it. Also, please add enumerateToAssignBlocksForUploadTask to the swift examples. They are there for the objc examples but not for swift. That being said, I'm still a little unclear how to use this with multipart uploads. AWSS3TransferUtilityMultiPartUploadTask does seem to have a taskIdentifier and I sometimes get cases where multiple progress blocks get "piled on each other".

So far I'm having some issues with crashes when doing tests with network connectivity loses and moving in and out of background mode. One thing I'm unclear on, does TransferUtility pause the multiparts on network loss or is that something we still need to do on our own? According to this it looks like it does not handle pausing https://github.com/aws/aws-sdk-ios/issues/891. I've tested pausing on my own with reachability but I'm still get crashes regarding putting null objects in a dictionary.

I'll try to file some bugs for the specific issues although I'm not sure how much longer they want me working on this as it's starting to seem like it's not much better than Transfer Manager.

scb01 commented 6 years ago

@jeffjvick

Sorry to hear that you are still facing issues. We fixed a couple of the crashes in 2.6.18. Could you try with that version - Also, I would appreciate it very much if you could file the specific issues and will work to get them resolved.

jeffjvick commented 6 years ago

@cbommas Thanks. I just tried 2.6.18 and it's definitely better. I think I've only seen it crash once so far but I'm still testing.

I've filed #924 as I'm not clear if we are required to suspend/resume tasks on network failure but from my testing it appears TransferUtility is doing this itself.

I am still having some issues with the file progress getting messed up when returning from the background. This was much worse prior to 2.6.18 (it would often crash) but now I'm still having issues with the progress not fulling hitting 1.0 when the upload completes. This may be due to my miss use of enumerateToAssignBlocks which is why I commented on #595.

I will try to file other issues as I discern a pattern.

scb01 commented 6 years ago

@jeffjvick

I will investigate the enumerateToAssignBlocks topic and get back to you on this thread.

jeffjvick commented 6 years ago

thanks @cbommas

Here is a copy of the code I made for enumerateToAssignBlocks for multipart uploads (I didn't see an exact example for multipart, just normal uploads).

   @objc func applicationDidBecomeActive(notification: NSNotification) {
        let transferUtility = AWSS3TransferUtility.default()

        transferUtility.enumerateToAssignBlocks(forUploadTask: {
            (task, progress, completion) -> Void in

            }, multiPartUploadBlocksAssigner: {
            (task: AWSS3TransferUtilityMultiPartUploadTask, progress, completion) -> Void in
                print("task " + task.transferID + " prog " + String(task.progress.fractionCompleted))
                print(self.progressBlock.debugDescription)

                let progressPointer = AutoreleasingUnsafeMutablePointer<AWSS3TransferUtilityMultiPartProgressBlock?>(&self.progressBlock)

                let completionPointer = AutoreleasingUnsafeMutablePointer<AWSS3TransferUtilityMultiPartUploadCompletionHandlerBlock?>(&self.completionHandler)

                // Reassign your progress feedback
                progress?.pointee = progressPointer.pointee

                // Reassign your completion handler.
                completion?.pointee = completionPointer.pointee

                // needed??
                //task.suspend()
                //task.resume()
        }, downloadBlocksAssigner: {
             (task, progress, completion) -> Void in
        })
    }

One thing I noticed is that AWSS3TransferUtilityMultiPartUploadTask does not have a taskIdentifier while AWSS3TransferUtilityUploadTask does. Why? I'm not sure if why I'm doing is correctly reassigning the progress and completion handlers as I sometime see the progress running at multiples of 1, 2, etc when coming out of the background.

jeffjvick commented 6 years ago

@cbommas Have you had a chance to look at the code snip I added above? Am I handling this correctly?

scb01 commented 6 years ago

@jeffjvick

Apologies for the delay in my response. There has been a bunch of confusion on about how to use the enumerateToAssignBlocks from other folks as well. So, I have a taken a crack at simplifying getting the latest status of a transfer, how to associate/re-associate progressBlocks and connectionHandlers to ongoing transfers in the latest version of the SDK (2.2.23).

I have added three convenience methods to the TransferUtilityTask

If the app transitions from background to foreground, you don't need to re-associate or setup connection handlers. The ones that you setup before the app went to the background will be in place and will work fine.

To re-associate connectionHandlers and progressBlocks after app restarts, I'd recommend the following approach (demonstrated for uploadTasks below)

 let uploadTasks = transferUtility.getUploadTasks().result
         for task in uploadTasks! {
             task.setCompletionHandler(completionHandler!)
             task.setProgressBlock(progressBlock!)
         }

You can repeat this for downloadTasks and MultipartUploadTasks.

Would love to get your feedback on how this works out for you.

scb01 commented 6 years ago

@jeffjvick

Regarding - "One thing I noticed is that AWSS3TransferUtilityMultiPartUploadTask does not have a taskIdentifier while AWSS3TransferUtilityUploadTask does"

The Mulitpart transfer mechanism uses a NSURLSessionTask per part to upload the file in pieces. That is the reason why there is no taskIdentifier on it. All tasks, however, have a unique TransferID, that you should use instead of taskIdentifier to keep track of transfers.

jeffjvick commented 6 years ago

@cbommas Thanks for these additions, I appreciate it. Adding status is great as that was something present in the UploadRequests of TransferManager that was not, up til now, present in TransferUtility. setProgressBlock should also be helpful as I was previously having to keep a list of expressions so I could later set the progress block in a different view but now I can just keep track of the tasks.

I will try this out and report back. Thanks.

SanCHEESE commented 6 years ago

@cbommas Ok, I've tried it out, but its not working. What am I doing wrong? When app terminates, I suspend all ongoing tasks using suspend. Then when app launches after creating AWSS3TransferUtility instance, I enumerate utility's getDownloadTasks.result tasks and there is none of them in returned array.

scb01 commented 6 years ago

@SanCHEESE

The transferUtility uses the value passed in for the key (see below) to link back to the transfers that were in progress in the previous run of the app

AWSS3TransferUtility.register(with: self.configurationAWSfor(video: video)!, forKey: key

Is the value of the key in your case a constant value? As a best practice, I would suggest is to register the TransferUtility in your app delegate. Once registered, you can look up the client wherever needed using the AWSS3TransferUtility.s3TransferUtility(forKey: key) method.

There is one nuance that I'm working on. The TransferUtility recovery mechanism happens asynchronously. So if you make the getDownloadTasks call immediately after registering the TransferUtility, there may be zero results based on timing ( there will essentially be 2 threads - the thread that the app is working on and the thread that is being used by the TransferUtility recovery mechanism).

In my testing so far, registering the TransferUtility in the app delegate and looking it up in a view controller subsequently seemed to always work; but I understand that this is a hit or miss approach. I will be working on a callback mechanism that will get a notification when the TransferUtility recovery is completed to address this in a more deterministic manner.

jeffjvick commented 6 years ago

@cbommas

I have tested setProgressBlock on previously created AWSS3TransferUtilityMultiPartUploadTask and it seems to work fine. This is a nice convince method as previously I had to keep track of all of my expressions so I could later attach a progressBlock to them. I have not yet test setCompletionHandler.

The auto re-assignment of progressBlocks when going in/out of background seems to work fine as well. I have not yet experienced any issues of mismatches when uploading multiple files at once.

I have noticed some issues with status though. For one thing, if you issue a cancel()to a AWSS3TransferUtilityMultiPartUploadTask while the network is down the status doesn't change to AWSS3TransferUtilityTransferStatusCancelled until the network is back up, it just stays in AWSS3TransferUtilityTransferStatusInProgress. Was this by design or a bug? TransferManager would sometimes get stuck if you tried to cancel while there was no network but I have not experienced that with TransferUtility.

Another issue I've run into is that now that TransferUtility handles pausing/resuming the upload when the network goes in and out it is not firing errors in the completionHandler like TransferManager would do. I was using these errors to change status bars regarding the upload being "paused" due to network. While I should be able to replace this with some Reachability code, I noticed that the status does not change to AWSS3TransferUtilityTransferStatusPaused, it's still AWSS3TransferUtilityTransferStatusInProgress. Is the paused status only being for manual suspends of uploads and not the network dropping? That's fine if so I just want to clarify this is not a bug.

jeffjvick commented 6 years ago

@cbommas @SanCHEESE

I also tested out transferUtility.getMultiPartUploadTasks().result.

When applicationWillTerminate fires I run though my list of mutlipartUploadTasks and suspend them all. Then when the app is restarted I check transferUtility.getMultiPartUploadTasks().result using let transferUtility = AWSS3TransferUtility.s3TransferUtility(forKey: "my-advanced-options"). Thank you @cbommas for pointing out that the key is needed but please update this in the docs as it's not clear.

I can see tasks in the array however since this is a multipart upload I often see more than one task for the same transferID. I'm not clear if I am supposed to resume all of these or would just one suffice? Also, when I start a multipart upload for the first time the completion handler normally just returns me one AWSS3TransferUtilityMultiPartUploadTask which I use to keep track of my uploads but in the case of this resuming, which of the many tasks should I pick? When I just pick the first one and try to setProgressBlock on it never updates the progress. Can you please offer more guidance on how to handle multipart for this case?

Also, I'm seeing the asynchronous nature of registering transferUtility as sometimes transferUtility.getMultiPartUploadTasks().result is empty when it should not be. I am registering transferUtility in the app delegate so I'm doing it as early as possible. Some type of notification appears necessary.

Honestly though I'm considering not using this since if a user force quits the app or powers off their phone while uploading they shouldn't really expect it to continue where they left off and as long as I'm not a memory hog I shouldn't get terminated while in the background.

SanCHEESE commented 6 years ago

@cbommas Still no luck. But sometimes there are download tasks in utility on app launch, but when I resume those downloads, download progress begins from 0, which means that already downloaded data was lost. I followed your advice and replaced [AWSS3TransferUtility defaultS3TransferUtility] with custom registered utility with constant key which is registered and restored using +S3TransferUtilityForKey. Knowing that utility registering is async I do wait until utility gets the tasks in bg thread:

NSTimeInterval totalWait = 0; NSArray *downloadTasks; while ((downloadTasks = utility.getDownloadTasks.result).count == 0 && totalWait < 5) { [NSThread sleepForTimeInterval:1]; totalWait += 1; } Sometime there are tasks, sometimes there are no. If there are, resuming download begins it anew, which I don't understand.

Let me explain what am I doing. First of all, I have 2 utilities for downloading. The first one is for normal downloads, second is for accelerated downloads. All utilities registered with constant keys. All the tasks created are retained and associated with downloading files and download path is simply a bucket key. All the tasks ongoing in the moment of termination are being suspended. When app is moved to an active state, utility's -(void)enumerateToAssignBlocksForUploadTask:downloadTask: is called.

scb01 commented 6 years ago

@jeffjvick

Let me address your points as best as I can

The auto re-assignment of progressBlocks when going in/out of background seems to work fine as well. I have not yet experienced any issues of mismatches when uploading multiple files at once._ Just to clarify, for this I am not auto-assigning anything. It is just that when the app goes into the background and transitions back to the foreground, the OS will make sure that all objects and references that were in memory before will remain as-is after. So it simply works :)

if you issue a cancel()to a AWSS3TransferUtilityMultiPartUploadTask while the network is down the status doesn't change to AWSS3TransferUtilityTransferStatusCancelled until the network is back up - I will look into this. The behavior is that the transfer task will go into a cancel status immediately, so there may be a bug.

Is the paused status only being for manual suspends of uploads and not the network dropping? Yes, the paused status is only for explicit suspends of uploads/downloads and not network drops. The TransferUtility will continue its attempt to transfer ( and retry as per the retry threshold) if the network drops intermittently.

I can see tasks in the array however since this is a multipart upload I often see more than one task for the same transferID. I'm not clear if I am supposed to resume all of these or would just one suffice? I haven't seen this in my local testing. The expectation is that you will only have one TransferUtilityTask per TransferID ( So only one task, no matter how many how many parts it has). There may be a bug and I will investigate and post back on it. If you can dump the contents of the array into a log and post the log here, it will be super helpful.

Some type of notification appears necessary. Agreed. I am working on this.

Honestly though I'm considering not using this since if a user force quits the app or powers off their phone while uploading they shouldn't really expect it to continue where they left off and as long as I'm not a memory hog I shouldn't get terminated while in the background. I see your point of view, though iOS may have a slightly different take based on how they have positioned background transfers. In any case, your strategy of suspending all transfers when applicationWillTerminate will enable you to fully control this as you see fit.

scb01 commented 6 years ago

@SanCHEESE

sorry about the troubles you are having on the timing issue. Unfortunately, the best solution for that is the callback (good news is that I'm working on it) as putting a delay can be hit or miss.

Why does the download start anew - I see that you are suspending the tasks when you get a signal that the app is going to be terminated. When I get the callback working, you will see all the tasks properly loaded up as the timing issue will be gone then.

For files that were being downloaded that were paused, if you resume them within a time limit ( typically 20 seconds based on a S3 server side timeout), it will continue where it left off. Otherwise, the transfer will error with a server time out and will be retried. For downloads, it will cause it to start from 0.

jeffjvick commented 6 years ago

@cbommas Thanks for the response

Her is the dump of transferUtility.getMultiPartUploadTasks().result. It seems to usually be five and they all appear to be the same:

savedMultipartUploadTasks   NSArray 5 elements  0x00000001c0241d70
[0] AWSS3TransferUtilityMultiPartUploadTask *   0x1c012b0e0 0x00000001c012b0e0
[1] AWSS3TransferUtilityMultiPartUploadTask *   0x1c012b0e0 0x00000001c012b0e0
[2] AWSS3TransferUtilityMultiPartUploadTask *   0x1c012b0e0 0x00000001c012b0e0
NSObject    NSObject    
_cancelled  BOOL    NO  false
_temporaryFileCreated   BOOL    NO  false
_retryCount int 0
_partNumber int 0
_transferID __NSCFString *  @"B9ADDF8F-8F76-4186-9FA8-D03BECB0F654" 0x00000001c0279180
_bucket __NSCFString *  @"test-upload"  0x00000001c005f3b0
_key    __NSCFString *  @"0F893573-B15F-407A-85DD-CAEA6D1DF9F7_76Kc8kuucDeFKirts"   0x00000001c008ec40
_progress   NSProgress *    0x1c012b180 0x00000001c012b180
_status AWSS3TransferUtilityTransferStatusType  AWSS3TransferUtilityTransferStatusPaused
_expression AWSS3TransferUtilityMultiPartUploadExpression * 0x1c005f380 0x00000001c005f380
_uploadID   __NSCFString *  @"YoSN8eN4gRHvbwc6VGc25Jz2iqhYbxEw4.AD_Ai7BNBO48LEAcJXbqaoUuCVyq9ZjOIrRF_BWvtWgDRJhdv3oKoQeMwBjmrJeHQ2ovZP8S0CbG6LAQdx462cTi44RRet9uBMgKop0FSIkKvP85VtSP3Ejrk4PIR6dB919mQn6hE-" 0x00000001c017a100
_waitingPartsDictionary __NSDictionaryM *   0 key/value pairs   0x00000001c0225420
_completedPartsDictionary   __NSDictionaryM *   0 key/value pairs   0x00000001c0225520
_inProgressPartsDictionary  __NSDictionaryM *   5 key/value pairs   0x00000001c0225480
_file   __NSCFString *  @"/private/var/mobile/Containers/Data/Application/1D76DF01-9470-4AC8-8B28-7C3091F594D3/tmp/F2FF6EEF-1F08-4BA5-8A1B-2CDFC83C329D.MOV"    0x00000001c012b040
_transferType   NSString *  nil 0x0000000000000000
_nsURLSessionID __NSCFString *  @"com.amazonaws.AWSS3TransferUtility.Identifier.transfer-utility-with-advanced-options" 0x00000001c00d6ff0
_databaseQueue  AWSFMDatabaseQueue *    0x1c0240030 0x00000001c0240030
_error  NSError *   nil 0x0000000000000000
_contentLength  __NSCFNumber *  (int)39211963   0xb000000025653bb2
[3] AWSS3TransferUtilityMultiPartUploadTask *   0x1c012b0e0 0x00000001c012b0e0
NSObject    NSObject    
_cancelled  BOOL    NO  false
_temporaryFileCreated   BOOL    NO  false
_retryCount int 0
_partNumber int 0
_transferID __NSCFString *  @"B9ADDF8F-8F76-4186-9FA8-D03BECB0F654" 0x00000001c0279180
_bucket __NSCFString *  @"test-upload"  0x00000001c005f3b0
_key    __NSCFString *  @"0F893573-B15F-407A-85DD-CAEA6D1DF9F7_76Kc8kuucDeFKirts"   0x00000001c008ec40
_progress   NSProgress *    0x1c012b180 0x00000001c012b180
_status AWSS3TransferUtilityTransferStatusType  AWSS3TransferUtilityTransferStatusPaused
_expression AWSS3TransferUtilityMultiPartUploadExpression * 0x1c005f380 0x00000001c005f380
_uploadID   __NSCFString *  @"YoSN8eN4gRHvbwc6VGc25Jz2iqhYbxEw4.AD_Ai7BNBO48LEAcJXbqaoUuCVyq9ZjOIrRF_BWvtWgDRJhdv3oKoQeMwBjmrJeHQ2ovZP8S0CbG6LAQdx462cTi44RRet9uBMgKop0FSIkKvP85VtSP3Ejrk4PIR6dB919mQn6hE-" 0x00000001c017a100
_waitingPartsDictionary __NSDictionaryM *   0 key/value pairs   0x00000001c0225420
_completedPartsDictionary   __NSDictionaryM *   0 key/value pairs   0x00000001c0225520
_inProgressPartsDictionary  __NSDictionaryM *   5 key/value pairs   0x00000001c0225480
_file   __NSCFString *  @"/private/var/mobile/Containers/Data/Application/1D76DF01-9470-4AC8-8B28-7C3091F594D3/tmp/F2FF6EEF-1F08-4BA5-8A1B-2CDFC83C329D.MOV"    0x00000001c012b040
_transferType   NSString *  nil 0x0000000000000000
_nsURLSessionID __NSCFString *  @"com.amazonaws.AWSS3TransferUtility.Identifier.transfer-utility-with-advanced-options" 0x00000001c00d6ff0
_databaseQueue  AWSFMDatabaseQueue *    0x1c0240030 0x00000001c0240030
_error  NSError *   nil 0x0000000000000000
_contentLength  __NSCFNumber *  (int)39211963   0xb000000025653bb2
[4] AWSS3TransferUtilityMultiPartUploadTask *   0x1c012b0e0 0x00000001c012b0e0
scb01 commented 6 years ago

@jeffjvick This is definitely a bug. I will fix it in an upcoming release. You are correct that all of these are references to the same object. You should only resume one of them.

scb01 commented 6 years ago

@jeffjvick The bug fixes for the issues you pointed out and the completion handler feature are included in the latest rev of the SDK. See my post on the https://github.com/aws/aws-sdk-ios/issues/759 thread to see how to setup the completion handler.

It will be great if you could give this a go and let me know your thoughts.

scb01 commented 6 years ago

@jeffjvick Just checking back to see if the fixes helped in resolving the issues you were encountering

jeffjvick commented 6 years ago

@cbommas I was able to try out the new version. I only see one task now instead of 5 so that appears to be fixed. Adding a completion handler onto the resumed task also works fine. There seems to be an issue with the progress block though as progress.fractionCompleted is always 1.0. Do you have an idea of why this may be occurring?

scb01 commented 6 years ago

@jeffjvick

Interesting. I haven't observed that in my testing. Is this happening for all types of transfers? Only after the transfer is resumed/continued after app restart?

jeffjvick commented 6 years ago

@cbommas It was only happening for resumes after app restart. I just tried the new 2.6.26 and it works fine. It looks like 2.6.25 has this fractionCompleted issue. Something must have been changed/fixed in 26.

I'm noticing another issue now. Do we still need to call AWSS3TransferUtility.interceptApplication(application, handleEventsForBackgroundURLSession: identifier, completionHandler: completionHandler) in func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @escaping () -> Void) {?

It seems when I do that the completion handler for a task is not firing when the upload completes in the background. If I omit the interceptApplication call though it works. Did something change because I thought that was required to notify Transfer Utility the background upload has completed.

scb01 commented 6 years ago

@jeffjvick You don't need to call the interceptApplication any more. With the recent changes in the TransferUtility, I have put in some measures to ensure that the completion handler is always fired when the transfer reaches an end state (i.e., completed successfully or ended up in an error)

jeffjvick commented 6 years ago

@cbommas

Okay that seems to align with what I’m seeing. You should probably update the documents about this and the many other things that have changed (like not needing to enumerate, etc)

I’m still having issues though where the completion handler does always fire when the upload finishes in the background. It seems to happen only when the debugger is not attached which makes debugging it difficult. Are there normal cases in which it will not fire?

jeffjvick commented 6 years ago

@cbommas

I'm still having an issues where background multipart uploading is not working when I'm NOT connected to the debugger and the upload time is longer than ~1 min. I think I'm not seeing it when the debugger is attached because the debugger never lets the app actually suspend: https://forums.developer.apple.com/message/42353#42353

I'm starting to think I never tested background uploading without the debugger connected. Do you have an idea of what may be causing this?

scb01 commented 6 years ago

@jeffjvick I am looking into this issue. I have made some changes to fix the resume issue for MultiPartUploads #1015 that should get out in the next release.

I think I can explain what you are seeing. For Multipart uploads, the TransferUtility uses a "relay" style architecture. The file to be uploaded is divided into 5MB chunks and uploaded as n requests in parallel. (n is determined by the multiPartConcurrencyLimit attribute in AWSTransferUtilityConfiguration). When one of the n requests is completed, then the next one i.e., n+1 is issued by the TransferUtility. And this process continues until all parts are uploaded and a final CompleteMultiPartUpload operation is executed to finalize the upload.

For the relay architecture to work, the app has to be running ( either in the foreground or background). If the app is closed/restarted, you will need to instantiate the TransferUtility and it will reattach to the ongoing transfers and continue them.

Based on your comments on this thread and in a couple of others, my understanding is that you are expecting the app to "woken up from background" or "restarted " when a transfer is completed. Is that right?

I'd like to jump on a call with you to discuss this further. Let me know if you are open to it and I can setup a conference call.

jeffjvick commented 6 years ago

@cbommas

Since you are using URLSession I was under the impression that urlSessionDidFinishEvents(forBackgroundURLSession session: URLSession) could be called to run our completionHandlers once all the parts have completed. But maybe there is something different about how a multi-part upload is handled that I'm not understanding.

Yes a call would be good, I'm free today, 8/31.

scb01 commented 6 years ago

@jeffjvick

How about 4:00 PM PST at the following conference bridge

United States Toll-Free: +1 855-552-4463 Meeting PIN: 9376 87 1968

jeffjvick commented 6 years ago

@cbommas That works, thanks.

jeffjvick commented 6 years ago

@cbommas Do you have an update on progress towards a version that allows multipart background uploads with more than n parts to complete?

scb01 commented 6 years ago

@jeffjvick

Thanks for your patience on this issue. I am working on this and am targeting to get this into the next rev of the SDK. I will post back on this thread once I have more info.

scb01 commented 6 years ago

@jeffjvick

Here is an update as promised. I did a bunch of tests using the TransferUtility and was able to consistently get uploads ( including multipart uploads) and downloads to complete successfully in the background. In my tests, I foregrounded and backgrounded the app multiple times and the transfer finished successfully every time.

My setup included

I had the following settings for the transferUtilityConfiguration

        transferUtilityConfiguration.retryLimit = 10 //I set this high to guard against server side timeout errors 
        transferUtilityConfiguration.multiPartConcurrencyLimit = 1 //I set it to 1 to ensure the relay style logic is triggered multiple times
        transferUtilityConfiguration.timeoutIntervalForResource = 15*60

The only change I had to make to my app was to add the following in the AppDelegate. 

func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @escaping () -> Void) {
        print ("handleEventsForBackgroundSession method called for session Identifier:" + identifier)
        AWSS3TransferUtility.interceptApplication(application, handleEventsForBackgroundURLSession: identifier, completionHandler: completionHandler)
    }

I think I may have added to the confusion on this topic by suggesting earlier that this callback was no-longer required. However, I was mistaken. Adding this callback to the App suggests to iOS that it should provide CPU cycles to the app when an ongoing NSURLSession task is completed. With this callback in place, I noticed that the OS automatically woke up the app in the background.

The speed of the upload/download varied over the course of the tests, depending on how the OS prioritized the app and associated transfers; however, the transfers finished successfully each time.

I was also able to find the issue in multipart uploads where the progress tracking was being under-reported when the app moves from background to the foreground. I have a fix for it, which I am hoping to get included in the next rev of the SDK.

Can you please try this on your side when you have a moment and let me know how it goes.

jeffjvick commented 6 years ago

@cbommas Thanks I will try this out.

I'm guessing it has to do with the interceptApplication part as I had removed that a while ago at your request. I thought I had tried a test with adding it back in but I was trying a lot of things at the time.

scb01 commented 6 years ago

@jeffjvick

Thank you. Please let me know how it goes. By the way, the latest rev of the SDK (2.6.31) contains the fix for the under-reporting of the mulitpart transfer progress bug

scb01 commented 6 years ago

@jeffjvick Just checking back. Let me know if you have any questions/issues that I can help with.

jeffjvick commented 6 years ago

@cbommas So I have been testing it out and I can see that the progress under-reporting is better (which was part of the reason I was thinking it wasn't working before) but I sometimes see it over-report by a few percent.

However, it still seems like the uploads are not continuing when going to the background or are possibly just progressing extremely slowly. I have see a few times where it finished in the background but usually it seems to have made minimal progress after many minutes of waiting.

I understand that iOS adjusts the priority of the background tasks but this seems excessive so I'm not sure if I am still missing something or is this really the speed at which background uploads take. My experience with other apps doing background downloads is that it is much faster.

scb01 commented 6 years ago

@jeffjvick

I ran a bunch more uploads using my testapp and the upload speeds were varying rather widely; I couldn't spot a pattern, other than what you observed - in some cases the transfer had completed, in other cases it had stalled. However, in all cases, the transfers did eventually finish, which led me to believe that the transfers were progressing extremely slowly.

I looked for information on how iOS schedules & prioritizes the transfers and I couldn't find anything definitive. What I have seen on the forums is it depends on a bunch of factors - is the device on Wifi or cellular, is the device fully charged, is it plugged in etc.

I also ran some tests by the setting the discretionary flag to Yes ( this is currently not set by the TransferUtility and defaults to No). With the discretionary flag set, it was actually even slower.

Regarding the progress over-reporting, is this showing up in your progress bars? I have seen that if you use a DispatchQueue.main.async to call your UI logic, it can have weird results from a sequencing perspective. I have had to implement logic as shown here (https://github.com/awslabs/aws-sdk-ios-samples/blob/0a301ad6bdf1cb2e1ecdda9354ea76daa9ea8555/S3TransferUtility-Sample/Swift/S3BackgroundTransferSampleSwift/UploadViewController.swift#L38) to prevent the progress bar from skipping forwards and backwards.

jeffjvick commented 6 years ago

@cbommas Thanks for looking at this. I had looked as well into how iOS priorities background transfers and like you said there is no definitive answer. I did read though that the isDiscretionary flag set to true is supposed to make things slower as you are telling iOS to leave it up to its discretion of how to prioritize your upload, which appears to always be "low" 😂. So that makes sense with what you saw when changing this.

https://developer.apple.com/documentation/foundation/nsurlsessionconfiguration/1411552-discretionary

One interesting thing to note though in that documentation is in the last paragraph: "For transfers started while your app is in the background, the system always starts transfers at its discretion". It's possible that when your scheduler starts new parts in the background the discretionary flag is getting set by iOS regardless of the initial setting and thus why it progresses so slowly in the background. This kind of make sense since I always see the first part(s) started in the foreground completing quickly in the background but then subsequent parts started in the background complete significantly slower.

I also did some tests using single part uploads and they all progress very quickly in the backend. This could also support the theory that the parts started in the background are getting hit with discretionary flag since in this the case there is only one "part" which is started in the foreground and thus has this flag set to false. You could try setting discretionary to true and doing a background single part upload and see if it is also slow

Could you check in your scheduler if this is possibly what was happening? (new parts getting started in the background have the discretionary flag set).

Regarding the progress over-reporting, I'm just doing a print(Float(progress.fractionCompleted)) in the progress block and I see it over-reporting. It seems to have to do with the file size, possibly the further the size is from a multiple of 5 the worse the over-reporting. For instance a 42.9Mb file ended with progress.fractionCompleted like so (using a concurrency of 1 part):

1.3723482
1.3752619
1.3781755
1.3810892
1.3840029
1.3869166
1.3898304
1.392744
1.3956577
1.3985714 <--- this is when it finished

Larger files exhibit less over-reporting, potentially since the ratio of the small difference in the one part is less. You might not notice it.