Issue with TusKit Uploads Pausing in Background and High Memory Usage

[nivinWorkflowlabs]

I'm encountering an issue with the TusKit iOS library where uploads pause when the app goes into the background and automatically resume when it returns to the foreground. To address this, I tried setting the chunk size to 0, which allowed the uploads to continue in the background. However, this approach caused the app to crash due to high memory usage when uploading files larger than 2GB.

Steps to Reproduce:

1. Implement resumable uploads using TusKit in an iOS app.
2. Set the chunk size to 0 in the TUSClient initialization.
3. Attempt to upload a file larger than 2GB.
4. Observe that the app crashes due to high memory usage.

Expected Behavior:

• The uploads should continue in the background without causing a memory-related crash.

Actual Behavior:

• The app crashes when attempting to upload files larger than 2GB with chunk size set to 0 due to high memory consumption.
• The app is not running in the background when I set the chunk size other than 0, eg: If I set the chunk size as 20MB, the app will upload in the background until 20MB. Then pauses and waits for the app to come in foreground to upload the next chunk.

Code Snippet:

func initializeTusClient(){
    do {
        let identifier = "com.workflowlabs.newsflowkmp.tus"
        let tusClient = try TUSClient(
            server: URL(string: Urls().uploadFile)!,
            sessionIdentifier: "TUS DEMO",
            sessionConfiguration: .background(withIdentifier: identifier),
            storageDirectory: URL(string: "/TUS")!,
            chunkSize: 0 // This causes high memory usage for large files
        )
        Self.uploadCommon = UploadCommonIosImpl(tusClient: tusClient)
        logDebug(message: "Upload initiated", tag: "UploadTusIos")
    } catch let error {
        logError(error: "Failed to create tusClient instance")
        logError(error: error)
    }
}

Environment:

• iOS version: 17.5.1
• TusKit version: 3.3.0
• Device: iPhone 11

Additional Context:

• Setting the chunk size to a non-zero value avoids the crash but results in the upload pausing when the app goes into the background.
• Any insights on how to balance background upload continuation with memory usage would be appreciated.

UploadCommonIosImpl.swift

import Foundation
import composeApp
import TUSKit
import OSLog

private let TAG = "UploadCommomIosImpl"
private let KEY_MEDIA_ID = "mediaId"
private let KEY_UPLOAD_SRC = "upload_source"
private let KEY_NAME = "filename"
private let KEY_TOKEN = "token"

let CHUNK_SIZE_IN_BYTES = 20000000 //20MB

class UploadCommonIosImpl:NSObject, UploadCommonIosInterface,TUSClientDelegate , URLSessionTaskDelegate {

    var tusClient:TUSClient!
    var status: ((UploadStatus) -> Void)! = nil
    let uploadRefMap:UploadRefMap = UploadRefMap()
    var updatedBaseUrl:URL? = nil

    func progressFor(id: UUID, context: [String : String]?, bytesUploaded: Int, totalBytes: Int, client: TUSKit.TUSClient) {
        let mediaId = uploadRefMap.getMediaId(uploadId: id)
        guard let mediaId = mediaId else {
            logDebug(message: "uploadProgress -> media Id is null for UUID \(id)", tag: TAG)
            return
        }
        dispatchProgress(mediaId:mediaId,totalBytes: Int64(totalBytes), bytesUploaded: Int64(bytesUploaded) )
    }

    private func dispatchProgress(mediaId:MediaId,totalBytes:Int64, bytesUploaded:Int64 ) {
        let updateDelay: Int64 = 1000
        let currentTimeMs = Int64(Date().timeIntervalSince1970 * 1000)
        var timeSizePair:(Time,Size)? = uploadRefMap.getLastTimeAndSize(mediaId: mediaId)
        if(timeSizePair == nil){ timeSizePair = (0,0) }
        let timeDifference = currentTimeMs - timeSizePair!.0
        if timeDifference < updateDelay { return }

        let bytesUploadedDub = Double(bytesUploaded)
        let totalBytesDub = Double(totalBytes)
        let progress = 100 * (bytesUploadedDub/totalBytesDub)
        let lastUploadedSize = timeSizePair!.1
        uploadRefMap.saveTimeAndSize(
            mediaId: mediaId,
            lastDispatchedTime: currentTimeMs,
            lastUploadedSize: bytesUploaded
        )
        let uploadProgress = UploadStatus.UploadProgress(
            mediaId: mediaId,
            totalBites: totalBytes,
            uploadedBites: bytesUploaded,
            dispatchedTimeDifference: timeDifference,
            previousUploadedBites: lastUploadedSize)
        status(uploadProgress)
        logDebug(message: "\(progress)% Uploading done for \(mediaId)", tag: TAG)
    }

    func didStartUpload(id: UUID, context: [String : String]?, client: TUSKit.TUSClient) {
        let mediaId = uploadRefMap.getMediaId(uploadId: id)
        guard mediaId != nil else {
            logDebug(message: "uploadStarted -> media Id is null for UUID \(id)",tag: TAG)
            return
        }
        logDebug(message: "UploadStarted for \(mediaId!)",tag: TAG)

        let uploadStarted = UploadStatus.UploadState
            .UploadStateStarted(mediaId: Int64.init(mediaId!))
        status(uploadStarted)
    }

    func didFinishUpload(id: UUID, url: URL, context: [String : String]?, client: TUSKit.TUSClient) {
        let mediaId = uploadRefMap.removeReference(uploadId: id)
        guard mediaId != nil else {
            logDebug(message: "uploadFinished -> media Id is null for UUID \(id)",tag: TAG)
            return
        }
        logDebug(message: "UploadFinished for \(mediaId!)",tag: TAG)

        let uploadFinished = UploadStatus.UploadState
            .UploadStateCompleted(mediaId: Int64.init(mediaId!))
        status(uploadFinished)
    }

    func uploadFailed(id: UUID, error: Error, context: [String : String]?, client: TUSKit.TUSClient) {
        let mediaId = uploadRefMap.removeReference(uploadId: id)
        guard mediaId != nil else {
            logDebug(message: "uploadFailed -> media Id is null for UUID \(id)",tag: TAG)
            return
        }
        logDebug(message: "UploadFailed for \(mediaId!), error: \(error)",tag: TAG)
        let uploadFailed = UploadStatus.UploadState
            .UploadStateError(mediaId: Int64.init(mediaId!))
        status(uploadFailed)
    }

    func fileError(error: TUSKit.TUSClientError, client: TUSKit.TUSClient) {
        //todo consider file error also
        logError(error: error)
    }

    func totalProgress(bytesUploaded: Int, totalBytes: Int, client: TUSKit.TUSClient) {
        // todo if needed add total progress of all uploads
    }

    init(tusClient:TUSClient){
        super.init()
        self.tusClient = tusClient
        tusClient.delegate = self
        tusClient.scheduleBackgroundTasks()
        let remainingUploads = tusClient.start()
        switch remainingUploads.count {
        case 0:
            print("No files to upload")
        case 1:
            print("Continuing uploading single file")
        case let nr:
            print("Continuing uploading \(nr) file(s)")
        }

        do{
            // When starting, you can retrieve the locally stored uploads that are marked as failure, and handle those.
            // E.g. Maybe some uploads failed from a last session, or failed from a background upload.
            let ids = try tusClient.failedUploadIDs()
            for id in ids {
                // You can either retry a failed upload...
                if try tusClient.retry(id: id) == false {
                    try tusClient.removeCacheFor(id: id)
                }
                // ...alternatively, you can delete them too
                // tusClient.removeCacheFor(id: id)
            }

            // You can get stored uploads with tusClient.getStoredUploads()
            let storedUploads = try tusClient.getStoredUploads()
            for storedUpload in storedUploads {
                print("\(storedUpload) Stored upload")
                print("\(storedUpload.uploadedRange?.upperBound ?? 0)/\(storedUpload.size) uploaded")
            }

        }catch{

            logError(error: error)
        }

    }

    func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: (any Error)?) {
        //todo
    }

    /// Uploads can fail while the app is in the background, so checking whether any upload got failed
    /// last time and updating the database with the error status for failed uploads
    private func checkAndUpdateFailedUploads(){
        do{
            let storedUploads = try tusClient.getStoredUploads()
            let mediaIdMap = mediaIdMapFromUploadInfos(uploadInfos: storedUploads)
            //todo

            let ids = try tusClient.failedUploadIDs()
            for id in ids {
                let mediaId = mediaIdMap[id]
                if(mediaId != nil) { status(UploadStatus.UploadStateError(mediaId: mediaId!)) }
            }
        } catch let error { logError(error: error)}
    }

    private func mediaIdMapFromUploadInfos(uploadInfos:[UploadInfo]) -> Dictionary<UploadId,MediaId> {
        var map:Dictionary<UploadId,MediaId> = Dictionary()
        uploadInfos.forEach { infos in
            let mediaId = Int64(infos.context![KEY_MEDIA_ID]!)
            map[infos.id] = mediaId
        }
        return map
    }

    func cancelAll(enableResume: Bool) {
        //todo use enable resume
        do{
            let tusClient = tusClient!
            tusClient.stopAndCancelAll()
            try tusClient.clearAllCache()
        }catch let error {
            logError(error: error)

        }
    }

    func cancelUploads(mediaIdList: [CancelRequest]) {
        mediaIdList.forEach { request in
            let id = uploadRefMap.removeReference(mediaId: request.mediaId)
            if(id != nil) { 
                do{
                    let tusClient = tusClient!
                    try tusClient.cancel(id: id!)
                    if !request.enableResuming {
                        logDebug(message: "Canceling upload for \(request.mediaId)", tag: TAG)
                        try tusClient.removeCacheFor(id: id!)
                    }else { logDebug(message: "Paused upload \(request.mediaId)", tag: TAG) }

                }catch let error{
                    logError(error: error)
                }
            }
        }
    }

    func listenToUploadStatus(status: @escaping (UploadStatus) -> Void) {
        self.status = status
        checkAndUpdateFailedUploads()
    }

    func releaseListener() {
        //todo release anything if there
    }

    func uploadAll(mediaList: [ProgressMediaEntity], accessTokenResponse: AccessTokenResponse) {
        mediaList.forEach { it in
            //todo check whether same file is being uploaded twise
            if(uploadRefMap.mediaIdExists(mediaId: it.mediaId)) { return }
            if(uploadRefMap.isLocked(mediaId: it.mediaId)) { return }

            uploadRefMap.lock(mediaId: it.mediaId)

            var uploadId = getUploadId(mediaId: it.mediaId)
            if uploadId == nil || !resumeUpload(uploadId: uploadId!) {
                // Either upload id is nil or resume got failed
                // So creates a new upload
                logDebug(message: "Uploading media \(it.mediaId)", tag: "temp")
                uploadId = uploadFile(media: it, accessToken: accessTokenResponse.access_token)
            }

            if(uploadId != nil){
                uploadRefMap.unlock(mediaId: it.mediaId)
                uploadRefMap.addReference(mediaId:it.mediaId, uploadId: uploadId!)
            }

        }
    }

    private func getUploadId(mediaId:MediaId) -> UploadId!{
        do{
            let storedUploads = try tusClient.getStoredUploads()
            for infos in storedUploads {
                let id = Int64(infos.context![KEY_MEDIA_ID]!)
                if(mediaId == id) { return infos.id }
            }
        }catch let error {
            logError(error: error)
        }
        logDebug(message: "Cannot find uploadId for mediaid: \(mediaId)", tag: TAG)
        return nil
    }

    private func resumeUpload(uploadId:UploadId) -> Bool{
        do {
            return try tusClient.resume(id: uploadId)
        } catch let error {
           logError(error: error)
           return false
        }
    }

    private func uploadFile(media:ProgressMediaEntity,accessToken:String) -> UploadId!{
        do{
            logDebug(message: "Requesting upload for \(media.mediaId )", tag: TAG)
            let url = URL.init(string: "file://\(media.cacheUri)")!
            let uploadId = try tusClient.uploadFileAt(
                filePath: url,
                uploadURL: updatedBaseUrl,
                context: getMetaData(accessToken: accessToken, media: media)
            )
            return uploadId
        }catch let error {
            status(UploadStatus.UploadState.UploadStateError(mediaId: media.mediaId))
            logError(error: error)
            return nil
        }

    }

    private func getMetaData(accessToken:String,media:ProgressMediaEntity) -> Dictionary<String,String>{
        var headers:Dictionary<String,String> = Dictionary()
        headers[KEY_TOKEN] = accessToken
        headers[KEY_NAME] = media.name
        headers[KEY_UPLOAD_SRC] = "iOS"
        headers[KEY_MEDIA_ID] = "\(media.mediaId)"
        return headers
    }

    func updateBaseUrl(url: String) {
        updatedBaseUrl = URL.init(string: url)
    }

}

iOSApp.swift

import SwiftUI
import TUSKit
import PhotosUI
import CoreLocation
import FirebaseMessaging
import FirebaseCore
import composeApp

class AppDelegate: NSObject, UIApplicationDelegate, MessagingDelegate {
    //Error 10.25.0 - [FirebaseMessaging][I-FCM002022] Declining request for FCM Token since no APNS Token specified
    private static var uploadCommon:UploadCommonIosImpl!

    func application(_ application: UIApplication,
                     didRegisterForRemoteNotificationsWithDeviceToken deviceToken: Data) {
        Messaging.messaging().setAPNSToken(deviceToken, type: MessagingAPNSTokenType.unknown)
    }

    func messaging(_ messaging: Messaging, didReceiveRegistrationToken fcmToken: String?) {
        logDebug(message: "tring to send token \(String(describing: fcmToken))",tag: "temp")
        guard fcmToken != nil else { return }

          let dataDict: [String: String] = ["token": fcmToken ?? ""]
          NotificationCenter.default.post(
            name: Notification.Name("FCMToken"),
            object: nil,
            userInfo: dataDict
          )
        sendFCMToServer?(fcmToken!)
    }

    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey : Any]? = nil) -> Bool {
        logDebug(message: "Configuring Firebase", tag:"temp")
        FirebaseApp.configure()

        UNUserNotificationCenter.current().delegate = self

        let authOptions: UNAuthorizationOptions = [.alert, .badge, .sound]
        UNUserNotificationCenter.current().requestAuthorization(
          options: authOptions,
          completionHandler: { _, _ in }
        )

        application.registerForRemoteNotifications()
        Messaging.messaging().delegate = self
        initializeTusClient()
        initializeIOSBridge()
        return true
    }

    // MARK: UISceneSession Lifecycle
    internal func application(_ application: UIApplication, configurationForConnecting connectingSceneSession: UISceneSession, options: UIScene.ConnectionOptions) -> UISceneConfiguration {
        // Called when a new scene session is being created.
        // Use this method to select a configuration to create the new scene with.
        return UISceneConfiguration(name: "Default Configuration", sessionRole: connectingSceneSession.role)
    }

    internal func application(_ application: UIApplication, didDiscardSceneSessions sceneSessions: Set<UISceneSession>) {
        // Called when the user discards a scene session.
        // If any sessions were discarded while the application was not running, this will be called shortly after application:didFinishLaunchingWithOptions.
        // Use this method to release any resources that were specific to the discarded scenes, as they will not return.
    }

    // https://developer.apple.com/documentation/uikit/uiapplicationdelegate/1622941-application
    internal func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @escaping () -> Void) {
        Self.uploadCommon.tusClient.registerBackgroundHandler(completionHandler, forSession: identifier)
    }

    func initializeIOSBridge(){
        SingletonsKt.iosBridge = IOSBridge(
            getUploadCommonIosInterface: { chunkSize in
                Self.uploadCommon
            },
            getUserId: { token, key in
                let value = JWTCommonSwift().getUserId(accessToken: token, userIdKey: key);
                return KotlinInt.init(integerLiteral: value)
            },
            filePreview:openFile,
            getConnectivity: { Connectivity() },
            getFFMpeg: { FFMpegIos() },
            setStatusBarColorIos:{ color in
                let uiColor = color as! UIColor
                UIApplication.shared.statusBarUIView?.backgroundColor = uiColor
            },
            passFcmSendBlock: { block in sendFCMToServer = block },
            initializeFCM: {fcmCommon in
                logDebug(message: "IOS initializing FCM", tag: "temp")
                Messaging.messaging().isAutoInitEnabled = true
                Messaging.messaging().token { token, error in
                  if let error = error {
                    print("Error fetching FCM registration token: \(error)")
                  } else if let token = token {
                    print("FCM registration token: \(token)")
                      fcmCommon.trySendFCM(token: token)
                  }
                }
            },
            getPhotoPickerViewController: {
                PhotoPickerViewController()
            }
        )

    }

    func initializeTusClient(){
        do{
            let identifier = "com.workflowlabs.newsflowkmp.tus"
            let tusClient = try TUSClient(
                server: URL(string: Urls().uploadFile)!,
                sessionIdentifier: "TUS DEMO",
                sessionConfiguration: .background(withIdentifier: identifier),
                storageDirectory: URL(string: "/TUS")!,
                chunkSize: CHUNK_SIZE_IN_BYTES
            )
            Self.uploadCommon = UploadCommonIosImpl(tusClient: tusClient)
            logDebug(message: "Chunk size \(CHUNK_SIZE_IN_BYTES)", tag: TAG)
            logDebug(message: "Upload initaited", tag: "UploadTusIos")
        }catch let error {
            logError(error: "Failed to create tusClient instance")
            logError(error: error)
        }
    }

}

var sendFCMToServer:((String) -> KotlinUnit)? = nil

@main
struct iOSApp: App {

    // register app delegate for Firebase setup
      @UIApplicationDelegateAdaptor(AppDelegate.self) var delegate
    init() {
        DiHelper_iosKt.doInitKoin(context: nil);
        SingletonsKt.userDataStore = CreateDataStore_iosKt.dataStore();
    }

    var body: some Scene {
        WindowGroup {
            ContentView()
        }
    }

}

extension UIApplication {
var statusBarUIView: UIView? {
    if #available(iOS 13.0, *) {
        let tag = 38482
        let keyWindow = UIApplication.shared.windows.filter {$0.isKeyWindow}.first

        if let statusBar = keyWindow?.viewWithTag(tag) {
            return statusBar
        } else {
            guard let statusBarFrame = keyWindow?.windowScene?.statusBarManager?.statusBarFrame else { return nil }
            let statusBarView = UIView(frame: statusBarFrame)
            statusBarView.tag = tag
            keyWindow?.addSubview(statusBarView)
            return statusBarView
        }
    } else if responds(to: Selector(("statusBar"))) {
        return value(forKey: "statusBar") as? UIView
    } else {
        return nil
    }
  }
}

extension AppDelegate: UNUserNotificationCenterDelegate {
  // Receive displayed notifications for iOS 10 devices.
  func userNotificationCenter(_ center: UNUserNotificationCenter,
                              willPresent notification: UNNotification) async
    -> UNNotificationPresentationOptions {
    let userInfo = notification.request.content.userInfo

    // With swizzling disabled you must let Messaging know about the message, for Analytics
    // Messaging.messaging().appDidReceiveMessage(userInfo)

    // Print full message.
    print(userInfo)

    // Change this to your preferred presentation option
        return [[.alert, .sound]]
  }

  func userNotificationCenter(_ center: UNUserNotificationCenter,
                              didReceive response: UNNotificationResponse) async {
    let userInfo = response.notification.request.content.userInfo

    // ...

    // With swizzling disabled you must let Messaging know about the message, for Analytics
    // Messaging.messaging().appDidReceiveMessage(userInfo)

    // Print full message.
    print(userInfo)
  }

}

info.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>CFBundleGetInfoString</key>
    <string></string>
    <key>BGTaskSchedulerPermittedIdentifiers</key>
    <array>
        <string>io.tus.uploading</string>
        <string>com.workflowlabs.newsflowkmp.tus</string>
        <string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
    </array>
    <key>CADisableMinimumFrameDurationOnPhone</key>
    <true/>
    <key>CFBundleDevelopmentRegion</key>
    <string>$(DEVELOPMENT_LANGUAGE)</string>
    <key>CFBundleDisplayName</key>
    <string></string>
    <key>CFBundleExecutable</key>
    <string>$(EXECUTABLE_NAME)</string>
    <key>CFBundleIdentifier</key>
    <string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
    <key>CFBundleInfoDictionaryVersion</key>
    <string>6.0</string>
    <key>CFBundleName</key>
    <string>$(PRODUCT_NAME)</string>
    <key>CFBundlePackageType</key>
    <string>$(PRODUCT_BUNDLE_PACKAGE_TYPE)</string>
    <key>CFBundleShortVersionString</key>
    <string>5.7.18</string>
    <key>CFBundleVersion</key>
    <string>37</string>
    <key>FirebaseMessagingAutoInitEnabled</key>
    <false/>
    <key>ITSAppUsesNonExemptEncryption</key>
    <false/>
    <key>LSApplicationCategoryType</key>
    <string>public.app-category.business</string>
    <key>LSMinimumSystemVersion</key>
    <string>14.0</string>
    <key>LSRequiresIPhoneOS</key>
    <true/>
    <key>NSAppTransportSecurity</key>
    <dict>
        <key>NSAllowsArbitraryLoads</key>
        <true/>
    </dict>
    <key>NSBluetoothAlwaysUsageDescription</key>
    <string>Bluetooth connection details is vital for uploading files</string>
    <key>NSCameraUsageDescription</key>
    <string>App uses Camera to take photos and videos for further modifications and upload to the main site</string>
    <key>NSContactsUsageDescription</key>
    <string>Contact details is vital for uploading files</string>
    <key>NSLocationWhenInUseUsageDescription</key>
    <string>Your current location is needed to filter search results</string>
    <key>NSHumanReadableCopyright</key>
    <string></string>
    <key>NSMicrophoneUsageDescription</key>
    <string>App uses Microphone to create audio recordings and video recordings for further modifications and upload to the main site</string>
    <key>NSMotionUsageDescription</key>
    <string>Motion details is vital for capturing videos</string>
    <key>NSPhotoLibraryUsageDescription</key>
    <string>App uses Photos Library to import media from the Photos app for further modifications and upload to the main site</string>
    <key>UIApplicationSceneManifest</key>
    <dict>
        <key>UIApplicationSupportsMultipleScenes</key>
        <false/>
    </dict>
    <key>UIBackgroundModes</key>
    <array>
        <string>fetch</string>
        <string>processing</string>
        <string>remote-notification</string>
    </array>
    <key>UILaunchScreen</key>
    <dict/>
    <key>UIRequiredDeviceCapabilities</key>
    <array>
        <string>armv7</string>
    </array>
    <key>UISupportedInterfaceOrientations</key>
    <array>
        <string>UIInterfaceOrientationPortrait</string>
        <string>UIInterfaceOrientationLandscapeLeft</string>
        <string>UIInterfaceOrientationLandscapeRight</string>
    </array>
    <key>UISupportedInterfaceOrientations~ipad</key>
    <array>
        <string>UIInterfaceOrientationPortrait</string>
        <string>UIInterfaceOrientationPortraitUpsideDown</string>
        <string>UIInterfaceOrientationLandscapeLeft</string>
        <string>UIInterfaceOrientationLandscapeRight</string>
    </array>
</dict>
</plist>

[donnywals]

Hi @nivinWorkflowlabs Thanks for reporting this!

I finally got around to investigating and it looks like we kept the file in-memory when we didn't need. PR #195 should fix this issue.

If possible, I'd love if you could give it a quick test run before I merge it into main