Improvement of docs on behavior in the background

781flyingdutchman / background_downloader

Flutter plugin for file downloads and uploads

Other

165 stars 76 forks source link

Improvement of docs on behavior in the background #402

Open escamoteur opened 1 month ago

escamoteur commented 1 month ago

Hi,

we are currently evaluating adding background uploading to our app. I think I read the readme quite careful but I was missing a clear statement on when callbacks are called and what happens in some situations. It would probably be a good idea to add the to the readme.

From what I read in one currently closed issue it seems that once the app is in the background no Dart code is called to notify about any changes. Is that correct? Which app live cycle states count as background in this case?
How are updates stored in the database when in the background? It seems using the data base is the only way to get informed if an upload was finished while in the background?
From what I read it seems that if the user kills the app any queued upload/download will be canceled, correct?
what happens if the app crashes or the OS kills the app?
will the last state of the upload/download in this case be stored in the database?
Does the app receive any status updates/callbacks once the app gets resumed by the user? do any callbacks get called when the app is restarted after it was killed while in the background?
is it possible to queue an HTTP call that is only executed when the upload was successful?

781flyingdutchman commented 1 month ago

If the app goes to the background (eg another app moves to the front, associated with "pause" on Android) everything continues as per normal for some time (determined by OS). If OS determines it needs resources it will suspend the app. Uploads will continue as normal, but we don't want to wake up the app for every status or progress update, so those are stored in a "local" database (using preferences). When your app resumes from suspend (so is "started" on Android) you need to establish your listeners or callbacks, then call resumeFromBackground. This will flush the locally stored database of status and progress updates and generate update events for each of those, that you need to listen to, to make sure your view of download/upload state is in sync with what happened while the app was suspended.

If the app crashes or the user kills the app (eg swipe away in app switcher) then by design the OS also kills outstanding and running background tasks.

Because of this, and because there is always a chance something gets missed, it is advisable for critical uploads/downloads to keep track of what you have queued and what has been completed in your own database, and use methods like allTasks to get a view on start-up of which tasks are still queued or running, and if necessary restart missing ones. You can leverage the downloader's database functionality for this (using trackTasks etc) and implement your own PersistentStorage so that your database and the one the downloader uses are truly the same (and eg use sqlite to back that database).

It is currently not possible to conditionally run tasks upon completion. There's a suggestion somewhere to add an option to wake up the app when a task completes, which might allow for this, but it's not currently in development.

Hope this helps, let me know if you have any other questions.

escamoteur commented 1 month ago

Hey, thanks a lot for this explanation. when does the database get updated? especially if I am able to add my own implementation this sounds as if it only happens when you call resumeFromBackground ? I think I got an idea of what I have to do to track all that. Thanks for pointing out `allTasks to query still running tasks.

so when the OS kills the app there won't be an entry in the local database but I have to check what the last successful entry was.

I was hoping that queueing a task that gets executed after a successful upload would be possible. for that no dart code would be necessary as we already know which data this request should contain but we only can make it if the upload succeeded first.

I guess adding some of this information to the readme would be a great help for anyone getting started with the package.

thanks for this package, I can only imagine the complexity getting and keeping this running

escamoteur commented 1 month ago

Btw you might want to think of republish the package with another name that signals that the package does uploads and downloads :-) I only found this one by accident

781flyingdutchman commented 1 month ago

Just to clarify, there are essentially two databases:

the "local" one (using preferences, on the native side of Android and iOS) which is used only to store updates that could not be posted to the main app (because it was suspended). Those are the ones you retrieve using resumeFromBackground, which triggers a flow of updates represented by the updates stored in the local database.
the database property of the FileDownloader(), which lives in Dart and the implementation of which can be changed by you, the developer, by implementing the PersistentStorage interface and passing your database implementation on your first call to the FileDownloader. This database contains TaskRecord entries for every task. To use the database (which is not active by default, as it adds overhead an many devs don't need to to maintain this state) you need to call trackTasks when your app starts. After this, every update that the app receives will trigger an update of the associated task's TaskRecord in the database. The default implementation uses the file system to keep this database, and I suggest you create a database using Itar or Hive or SQLite for your file uploading needs, and implement the PersistentStorage interface such that that same database is also used by the FileDownloader. That way, things stay in sync, and you can use this database for instance at startup, after processing resumeFromBackground to compare what your database thinks still needs to be uploaded with what allTasks says it is still working on - any discrepancy you can then fix.

so when the OS kills the app there won't be an entry in the local database but I have to check what the last successful entry was.

There will be entries in the local database, up to the point where the OS killed the upload, and this is exactly why you need the logic I described above: when you next call resumeFromBackground it will show that the upload is running, even though it was killed and will never complete. By checking allTasks you will see that the FileDownloader says that task is NOT running, which tells you that you need to re-enqueue it (and remove the zombie entry from the database).

I was hoping that queueing a task that gets executed after a successful upload would be possible.

It's difficult to generalize this, which is why I think the better solution is to wake up the app when a task finishes (as a configurable option for those that need this) so you can then do whatever you need to do. Waking up the app from within Android or iOS is a bit complicated though, so I haven't gotten to that.

I guess adding some of this information to the readme would be a great help for anyone getting started with the package.

I've tried to explain it in the databases section, but clearly not well enough :). It's already a long README though.

thanks for this package, I can only imagine the complexity getting and keeping this running

Appreciated - it has definitely grown into a bit of a monster

Btw you might want to think of republish the package with another name that signals that the package does uploads and downloads :-) I only found this one by accident

As is typical, this started as a very simple background downloader, then got requests to add notifications, a database, uploads, then parallel downloads, multi-part uploads, data tasks, etc - it keeps growing, and the name hasn't kept up but it's impossible to change now.

781flyingdutchman commented 4 weeks ago

It is currently not possible to conditionally run tasks upon completion. There's a suggestion somewhere to add an option to wake up the app when a task completes, which might allow for this, but it's not currently in development.

I've actually made some progress here, and would like to ask your help in testing it. If you check out the callbacks branch of this repo, I have implemented (for now for iOS only, but working on Android) an option to configure a task to call a callback just before and/or just after the task runs. This would allow you to modify certain task properties before it runs (e.g., refresh an expired auth token) or to do something upon task completion.

To use this, add an option property to the task, which itself is a TaskOption that takes a onTaskStart or onTaskFinished constructor argument. You must supply that callback function, and it will get called at the appropriate time.

What is especially difficult to test is what happens when a task has been waiting or running a long time, and the app has been suspended by the time the callback is called. This requires waking up the Dart side, and that is done differently in iOS and Android - which is why I would like to ask if you can test that this works under load. One specific thing I am unsure about is that as I understand it on Android only we will 'wake up' a headless version of the FlutterEngine (i.e. no UI attached - and unclear if it starts at main()) so I suspect that the function you pass as a callback needs to be (or is preferably, not sure!) a simple function that does not fully rely on your app's state (as again, my understanding is that the app itself may not be running when the callback is called). I think it is conceptually as if your callback is called within a background isolate, not the main isolate. This 'simple function' can make a direct HTTP call, but may not be able to do more complex things such as scheduling another DownloadTask using the FileDownloader() - I'd love feedback on what you find does and does not work, first for iOS and in a few days I will follow up with Android.

Let me know if you can help!

escamoteur commented 3 weeks ago

Hey, that sounds pretty amazing. I m currently adding your package to our app and could easily add such a callback. Unfortunately I don't know how on Android the app is woken. Did you check how the background_fetch package is doing it? Am 28. Okt. 2024, 22:55 +0100 schrieb 781flyingdutchman @.***>:

It is currently not possible to conditionally run tasks upon completion. There's a suggestion somewhere to add an option to wake up the app when a task completes, which might allow for this, but it's not currently in development. I've actually made some progress here, and would like to ask your help in testing it. If you check out the callbacks branch of this repo, I have implemented (for now for iOS only, but working on Android) an option to configure a task to call a callback just before and/or just after the task runs. This would allow you to modify certain task properties before it runs (e.g., refresh an expired auth token) or to do something upon task completion. To use this, add an option property to the task, which itself is a TaskOption that takes a onTaskStart or onTaskFinished constructor argument. You must supply that callback function, and it will get called at the appropriate time. What is especially difficult to test is what happens when a task has been waiting or running a long time, and the app has been suspended by the time the callback is called. This requires waking up the Dart side, and that is done differently in iOS and Android - which is why I would like to ask if you can test that this works under load. One specific thing I am unsure about is that as I understand it on Android only we will 'wake up' a headless version of the FlutterEngine (i.e. no UI attached - and unclear if it starts at main()) so I suspect that the function you pass as a callback needs to be (or is preferably, not sure!) a top-level, simple function that does not fully rely on your app's state (as again, my understanding is that the app itself may not be running when the callback is called). This 'simple function' can make a direct HTTP call, but may not be able to do more complex things such as scheduling another DownloadTask using the FileDownloader() - I'd love feedback on what you find does and does not work, first for iOS and in a few days I will follow up with Android. Let me know if you can help! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

781flyingdutchman commented 3 weeks ago

Great, thanks! I am especially interested in understand under what circumstances (if any) the callback is NOT called when it is supposed to. And good suggestion - I will check out that package.

escamoteur commented 3 weeks ago

Only that I mostly develop using android. So it would be good to have the android version. Does the callback have access to the result of the uploadtask? Am 29. Okt. 2024, 23:36 +0100 schrieb 781flyingdutchman @.***>:

Great, thanks! I am especially interested in understand under what circumstances (if any) the callback is NOT called when it is supposed to. And good suggestion - I will check out that package. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

781flyingdutchman commented 3 weeks ago

Ah, ok. Working on that, but may be a few days - no hurry, I'll ping this thread when it's available. And yes, the callback passes the last TaskStatusUpdate so everything you get when a task finishes for whatever reason that is normally sent to the listener will be available in the callback. The onTaskStartCallback has the task itself as a parameter, and allows your callback to return a modified task. This enables for example refreshing an expired auth token before that becomes an issue - the returned task will have the refreshed auth headers and that is what will be used for the request. I plan to automate that a bit further by adding an Authentication object as an option for the TaskOptions that allows the native code to determine whether refresh is required, so that the app does not have to call a callback for every task - but that's a later addition.

If you have feedback on the design of the TaskOptions API (including the callbacks) then let me know, as this is the moment when changes are still possible.

781flyingdutchman commented 3 weeks ago

OK, the latest callbacks branch has this implemented for Android, iOS and desktop. From the readme:

OnTaskStart and OnTaskFinished "native" callbacks

For more complex situations you can use OnTaskStart and OnTaskFinished callbacks. This is only required if you need to - for example - refresh an expired auth token just before an enqueued task starts, or conditionally call your server to confirm an upload has finished successfully, which requires the callback to be called even when the main application has been suspended by the OS. To add a callback to a Task, set its options property, e.g. to add an onTaskStart callback:

final task = DownloadTask(url: 'https://google.com',
   options: TaskOptions(onTaskStart: myStartCallback));

where myStartCallback must be a top level or static function.

For most situations, using the event listeners or registered "regular" callbacks is recommended, as they run in the normal application context on the main isolate. Native callbacks are called directly from native code (iOS, Android or Desktop) and therefore behave differently:

Callbacks are called even when an application is suspended
On iOS, the callbacks runs in the main isolate
On Android, callbacks run in a shared background isolate, though there is no guarantee that every callback shares the same isolate as another callback
On Desktop, callbacks run in the same isolate as the task, and every task has its own isolate

You should assume that the callback runs in an isolate, and has no access to application state or to plugins. Native callbacks are really only meant to perform simple "local" functions, operating only on the parameter passed into the callback function.

OnTaskStart

Callback with signatureFuture<Task?> Function(Task original), called just before the task starts executing. Your callback receives the original task about to start, and can modify this task if necessary (for example to refresh an auth token). If you make modifications, you return the modified task - otherwise return null to continue execution with the original task. You can only change the task's url (including query parameters) and headers properties - making changes to any other property may lead to undefined behavior.

OnTaskFinished

Callback with signature Future<void> Function(TaskStatusUpdate taskStatusUpdate), called when the task has reached a final state (regardless of outcome). Your callback receives the final TaskStatusUpdate and can act on that.

philitell commented 3 weeks ago

@781flyingdutchman Thank you for the great explanations in this thread!!!

781flyingdutchman commented 2 weeks ago

@escamoteur have you had a chance to take a look at the callbacks mechanism (in Android)? I'm working on adding an onAuth callback to help with token refresh for request authentication, using the same mechanism, so would value feedback on how reliable to callback mechanism is.

escamoteur commented 2 weeks ago

Sorry was too caught up in a huge refactoring and setting up the new Flutter forum forum.itsallwidgets.com probably will be on it next week Am 7. Nov. 2024, 21:36 +0100 schrieb 781flyingdutchman @.***>:

@escamoteur have you had a chance to take a look at the callbacks mechanism (in Android)? I'm working on adding an onAuth callback to help with token refresh for request authentication, using the same mechanism, so would value feedback on how reliable to callback mechanism is. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

781flyingdutchman commented 2 weeks ago

Adding authorization using the same mechanism, on the auth branch of the repo until published. Would like feedback on how this is working.

Authorization

The Auth object (which can be set as the auth property in TaskOptions) contains several properties that can optionally be set:

accessToken: the token created by your auth mechanism to provide access. It is typically passed as part of a request in the Authorization header, but different mechanisms exist
accessHeaders: the headers specific to authorization. In these headers, the template {accessToken} will be replaced by the actual accessToken property, so a common value would be {'Authorization': 'Bearer {accessToken}'
accessQueryParams: the query parameters specific to authorization. In these headers, the template {accessToken} will be replaced by the actual accessToken property
accessTokenExpiryTime: the time at which the accessToken will expire.
refreshToken, refreshHeaders and refreshQueryParams are similar to those for access (and replace the {refreshToken} template)
refreshUrl: url to use for refresh, including query parameters not related to the auth tokens
onAuth: callback that will be called when token refresh is required

The downloader uses the auth object on the native side as follows:

Just before the task starts, we check the accessTokenExpiryTime
If it is close to this time, the downloader will call the onAuth callback (your code) to refresh the access token
- A defaultOnAuth function is included that calls auth.refreshAccessToken using a common approach, but use your own onAuth callback if your auth mechanism differs
- The Task returned by the onAuth call can change the Auth object itself (e.g. replace the accessToken with a refreshed one) and those values will be used to construct the task's request
The Task request is built as follows:
- Start with the headers and query parameters of the original task. You should have all headers and query parameters that are not related to authentication here
- Add or replace every header and query parameter from the accessHeaders and accessQueryParams to the task's headers and query parameters, substituting the templates for accessToken and refreshToken
- Construct the task's server request using these merged headers and query parameters

A typical way to construct a task with authorization and default onAuth refresh approach then is:

final auth = Auth(
    accessToken: 'initialAccessToken',
    accessHeaders: {'Authorization': 'Bearer {accessToken}'},
    refreshToken: 'initialRefreshToken',
    refreshUrl: 'https://your.server/refresh_endpoint',
    accessTokenExpiryTime: DateTime.now()
            .add(const Duration(minutes: 10)), // typically extracted from token
    onAuth: defaultOnAuth // to use typical default callback
);
final task = DownloadTask(
    url: 'https://your.server/download_endpoint',
    urlQueryParameters: {'param1': 'value1'},
    headers: {'Header1': 'value2'},
    filename: 'my_file.txt',
    options: TaskOptions(auth: auth));

781flyingdutchman commented 1 week ago

Hi, just checking in if you've had a chance to test the callbacks (perhaps including Auth).

escamoteur commented 1 week ago

Hi really sorry too busy with project (background upload is only the next requirement) and maintaining my own projects. Am 18. Nov. 2024, 20:31 +0100 schrieb 781flyingdutchman @.***>:

Hi, just checking in if you've had a chance to test the callbacks (perhaps including Auth). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>