Open tonysung opened 7 years ago
In the future, we plan on adding an explicit "upload/download in the background" as opposed to simply going with the OS. Adding a flag for this isn't particularly difficult: the hard part is fetching backgrounded events and re-attaching progress listeners (as the ones added before the app was backgrounded disappear once backgrounded). We have a good way to solve this, but (somewhat surprisingly) haven't had significant requests for this from developers.
The API might look something like:
// Enable background uploads
[FIRStorage setBackgroundTransfersEnabled: YES];
// Fetch backgrounded uploads and re-attach observers
[storage backgroundedUploadTasksWithBlock:^(UploadTask *task){
[task observeStatus:FIRStorageTaskStatusProgress
handler:^(FIRStorageTaskSnapshot *snapshot) {
// A progress event occurred
}];
}];
Thoughts?
Yes this is good.
Just two requests if this is going to be implemented:
How about getting rid of a global flag and creating individual methods for background uploads and downloads?
[ref putData:data inBackgroundWithCompletion:^(FIRStorageMetadata *metadata, NSError *error){
// upload occurs in background
}];
[ref writeFile:localURL inBackgroundWithCompletion:^(NSURL *localFile, NSError *error) {
// download occurs in background
}];
I think this ends up being more discoverable as well.
Note that we'd still have to offer the methods to fetch backgrounded tasks.
This is definitely better.
Are there any updates on this?
@berkcoker unfortunately nothing, as a majority of dev efforts are going into Firestore at the moment. Happy to review a PR. GTMSessionFetcher supports background uploads/downloads (https://github.com/google/gtm-session-fetcher/blob/master/Source/GTMSessionFetcher.h#L79-L115), so it's not terribly difficult to add (honestly, we're more blocked on implementing it on Android and JS).
@mcdonamp I too would like background uploads, especially support for offline handling that automatically resumes when the device is online. E.g. upload a photo for a post, then insert that post into Firestore when a device or app comes back online. Firebase does claim to support offline capabilities for most products, I was surprised to find the same ethos doesn't currently apply to Firebase Storage.
I wonder if there are any updates on this? I'm currently using signed URLs, which I found easier to implement, but this feature feels like a no-brainer
There are unfortunately no updates at this point. We will update this issue when this changes.
Background uploads shouldn't be (only?) manually performed like suggested in https://github.com/firebase/firebase-ios-sdk/issues/147#issuecomment-318427481
IMO in most of the case, if an uploadTask
is uploading while the user is leaving the app in background, a background upload should be automatically triggered (if enabled by a flag in the uploadTask
).
Found this issue when searching on how to set the NSURLSessionConfiguration.discretionary
property to true
, because the OS default is false
. How would someone go about that?
I also got confused a bit at this point, because @tonysung mentions that true
is the default, but in reality it is false
, therefore exactly how he wanted it. This may have changed in the meantime, as he hasn't commented on this since.
Really would love to see this implemented soon!
Anyone know of any work arounds while it's not supported directly in the SDK?
Sure, here's how I do it. Please let me know if you have any trouble.
class PhotoUploadManager {
static var urlSessionIdentifier = "photoUploadsFromMainApp" // Should be changed in app extensions.
static let urlSession: URLSession = {
let configuration = URLSessionConfiguration.background(withIdentifier: PhotoUploadManager.urlSessionIdentifier)
configuration.sessionSendsLaunchEvents = false
configuration.sharedContainerIdentifier = "my-suite-name"
return URLSession(configuration: configuration)
}()
// ...
// Example upload URL: https://firebasestorage.googleapis.com/v0/b/my-bucket-name/o?name=user-photos/someUserId/ios/photoKey.jpg
func startUpload(fileUrl: URL, contentType: String, uploadUrl: URL) {
Auth.auth().currentUser?.getIDToken() { token, error in
if let error = error {
print("ID token retrieval error: \(error.localizedDescription)")
return
}
guard let token = token else {
print("No token.")
return
}
var urlRequest = URLRequest(url: uploadUrl)
urlRequest.setValue("Bearer \(token)", forHTTPHeaderField: "Authorization")
urlRequest.setValue(contentType, forHTTPHeaderField: "Content-Type")
urlRequest.httpMethod = "POST"
let uploadTask = PhotoUploadManager.urlSession.uploadTask(with: urlRequest, fromFile: fileUrl)
uploadTask.resume()
}
}
}
I should note that the comment about app extensions and the sharedContainerIdentifier field are from code I wrote ages ago, when my app had an extension, so use at your own risk.
Any update on this?
@OkiRules Sorry, no update available. We're happy to review a PR.
Would also like to know if there's been any updates on this. @Ruberik how did you construct the upload url? I've tried to follow the example url you posted but doesn't seem to work.
@mhle Let's say your bucket name is FOO, and you want the file to go in bucket FOO at the path BAR/BAZ/QUX.jpg. Then you'd upload it to: https://firebasestorage.googleapis.com/v0/b/FOO/o?name=BAR/BAZ/QUX.jpg
You'll need to set up your storage rules to allow uploads to that location.
Thanks! that worked, is there a way to get notified about progress and completion inside the app? I tried conforming to URLSessionTaskDelegate and implement urlSession(session:task:didSendBodyData:totalBytesSent:totalBytesExpectedToSend:) but didn't get any callback.
On Sat, Apr 20, 2019 at 3:32 AM Bartholomew Furrow notifications@github.com wrote:
@mhle https://github.com/mhle Let's say your bucket name is FOO, and you want the file to go in bucket FOO at the path BAR/BAZ/QUX.jpg. Then you'd upload it to: https://firebasestorage.googleapis.com/v0/b/FOO/o?name=BAR/BAZ/QUX.jpg
You'll need to set up your storage rules to allow uploads to that location.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-ios-sdk/issues/147#issuecomment-484980230, or mute the thread https://github.com/notifications/unsubscribe-auth/AAE4TM5GXXYPIYR7PTBHAW3PRIF23ANCNFSM4DTXORSA .
nvm, I got it to work, many thanks for your help! I hope we'll get an official solution via the sdk soon.
@mhle I also really hope it is part of the SDK soon, it would be very helpful to not have to make all the special calls... it should already be integrated through the SDK.
We are discussing internally to see if we can allocate resource to this feature request. We are sorry that we don't have a better update at this time.
@schmidt-sebastian any update?
Hi @schmidt-sebastian @morganchen12, there are any updates? I would like to make it possible to use my app even when the user is offline/have a bad signal, the Real time database knows how to handle those situations, but using Firebase Storage making it unhelpful, 'cause at 80% of cases the user is uploading an image with the data that sent to the database.
No progress on this issue unfortunately. You can work around it by generating a download URL and using NSURLSession's background transfer methods.
@morganchen12 Tried to generate download URL (using downloadURL(completion)
) as you suggested, but seems like I can't do so if the file haven't uploaded yet.
May you give me some way to get the download URL that the file would be uploaded to? (I need to save also the token that coming with this URL to be able to show the image in my app, didn't find a way to generate it manually).
What I'm actually trying to do is to use the image Data
that I'm uploading with putData()
as cache to the data that will come from the url (that would be exactly the same data), while I'll upload the data to Firebase (with NSURLSession, as you suggested), so I'll be able to save the url into Firebase Database (that supports offline mode and also is much faster 'cause it's just uploading text) and in the meanwhile show the cached image to the user.
You should consider using a Cloud Function to do the upload and database update to guarantee update atomicity. The Cloud Function can be an https callable url that you transfer data to via NSURLSession.
@morganchen12 - If we use cloud function then in that case cloud function has limitations and it will not allow you to upload large file for example 25MB Video file.Best solution is to make it possible via upload/download in background.
@shahmaulik That’s depends on your needs. I was able to make this workaround with Firebase functions ’cause I don’t upload files bigger than 5MB.
Yet, I didn’t close this issue ‘cause I think that they should add some build-in background upload option, also to support situations like you’ve described here.
If this is a question of resource allocation, then where is the right place for developers like us to cast a "vote" so you can see which FRs are the ones we want the most? Once I was told to +1 something on some google bug reporter site but I couldn't figure out how to -- the bug reporter site was harder to use than the Firebase iOS SDK by a million miles 😄
@xaphod - +1 to the original post of this issue - scroll to the top here and see the thumbs-up (currently at 17)
Since this is an iOS SDK only feature request and not a feature request for the Firebase Storage backend product, this GitHub issue is probably the best way to raise attention.
Color me confused: it turns out there is background upload support already. I noticed this when I restarted my app and saw GTMSessionUploadFetcher restoring upload fetcher
, and watched somewhat gob-smacked as a file that had earlier failed to upload, suddenly uploaded. I googled around but found no mention of this functionality anywhere -- is this missing from the documentation? There's nothing in the iOS firebase storage docs that I could see.
I did some further digging and found that in GTMSessionFetcher.m
, currently line 658, one sees self.usingBackgroundSession = YES;
, which gets (eventually) hit when I call storageRef().putFile(from: dataURL, metadata: metadata)
. This of course completely conflicts with the retry logic I wrote...
Is this new?
... update: I made a draft PR (for discussion purposes) that shows how i'm disabling GTMSessionFetcher's behavior, as it conflicts with my retry/offline logic when things complete in the background. That's because FIRStorage does not reconnect to GTMSessionFetcher's background sessions so my app never gets notified if something becomes a background session task, then fails/succeeds etc. https://github.com/firebase/firebase-ios-sdk/pull/6052
@xaphod There are two distinct scenarios here:
1) An upload that continues while the app is running in the background. This is something I would like to support via a first party API, but we haven't been able to staff this effort.
2) An upload that is restarted when the app is restarted (which seems to be the problem you are running into). I don't believe the SDK should retry these, as this would lead us into an area where we have a very unreliable offline cache and brings up all sorts of authentication issues. I am surprised that this is how GTMSessionUploadFetcher
works.
Ideally, we would find a way to staff (1) and prohibit (2). Your change prohibits (2) but requires an API change that would likely be obsolete once (1) lands.
Hi Sebastian,
Thanks for the thoughtful reply.
There appears to be code in GTMSessionFetcher that covers #1 (or at least a part of it) too - see GTM_BACKGROUND_TASK_FETCHING
Re 2 I agree that retries here make no sense. IMO FIRStorage should either “hook up” to GTMSessionFetcher’s background URLSession in a way that apps on top can become aware when background xfers complete/fail, or, as you say, disable this functionality by default (what I’m currently doing). To me it makes no sense that some transfers silently continue after the app terminates without the app explicitly opting into this behavior. I found one user on StackOverflow who got bitten by this.
But for giant files, hooking up to GTMSessionFetcher would be the preferred method ie for large video uploads...
On Wed, Jul 15, 2020 at 7:56 PM Sebastian Schmidt notifications@github.com wrote:
@xaphod https://github.com/xaphod There are two distinct scenarios here:
- An upload that continues while the app is running in the background. This is something I would like to support via a first party API, but we haven't been able to staff this effort.
- An upload that is restarted when the app is restarted (which seems to be the problem you are running into). I don't believe the SDK should retry these, as this would lead us into an area where we have a very unreliable offline cache and brings up all sorts of authentication issues. I am surprised that this is how GTMSessionUploadFetcher works.
Ideally, we would find a way to staff (1) and prohibit (2). Your change prohibits (2) but requires an API change that would likely be obsolete once (1) lands.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-ios-sdk/issues/147#issuecomment-659074852, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4Z6LQ2FXMYUOBRN74YFYLR3Y6Z5ANCNFSM4DTXORSA .
-- Tim Carr Founder, Solodigitalis M: 289-237-1935 W: solodigitalis.com A: 270 Sherman Ave N, Hamilton, ON L8L 6N4
@thomasvl can likely comment on that code, given that he wrote it way back when (https://github.com/google/gtm-session-fetcher/commit/f9ab2db3dc46af0f727090376f67d69067efbdc2 is the most recent git blame).
I find that behavior surprising (we never set the fetcher.useBackgroundSession = YES
flag, though I vaguely remember toying with it back in the day), and I am fairly certain it is not due to the useBackgroundSession
being set, but rather there being a prior session identifier. I assume what's happening is not that the file is being uploaded in the background, but rather that it's being re-started after the failed foreground upload, and completing in the foreground.
You can likely test this by uploading a large file and force killing the app, then seeing if it gets restored (admittedly I don't remember if GTMSessionFetcher persists session info, but I don't think it does).
On the API surface, there are a few proposals: https://github.com/firebase/firebase-ios-sdk/issues/147#issuecomment-318193709. Personally I'm a fan of per-upload/download calls, and then a way of fetching all in progress tasks. The main issue from a usability standpoint is that you'll have to add progress handlers twice (and potentially de-dupe the events, though I'm sure there's a way we could probably avoid firing callbacks twice).
There's also the API I snuck in a long time ago and never had time to implement that would hook into the GTMSessionFetcher stuff and re-hydrate those tasks: https://github.com/firebase/firebase-ios-sdk/blob/master/FirebaseStorage/Sources/FIRStorage.m#L249-L263
Thanks for the history & context, that's helpful.
The reason I put together the draft PR is because I am seeing background
sessions created when calling ref.putFile(url:). Specifically on
GTMSessionFetcher.m line 646 where it creates the URLSession, this code is
being hit:
_configuration = [NSURLSessionConfiguration backgroundSessionConfigurationWithIdentifier:sessionIdentifier];
-- right
after calling putFile. Stack trace at end of email.
My understanding is that this means that the uploads can finish without my
completion handler from putFile ever running, which is a problem for my
app's retry logic.
You don't have to set useBackgroundSession to YES for this to happen (you
are correct: I never saw that get set to YES) - these background sessions
get created automatically for non-data (upload) tasks. At least they do for
me, in GTMSessionFetcher 1.4.0
All of the approaches you linked look good to me. I'm of the opinion that the most important thing is that the API can guarantee to make at least one callback for every attempted upload/download, so that the app can perform retries if possible. In other words, I value delivery-guarantee over speed-of-delivery and i'm happy to sacrifice background (out of app) transfers to get it. Wouldn't be the end of the world (to me) if caller had to be idempotent and callbacks could fire more than once (might make task rehydration simpler?).
Edit: didn't realize email replies can't handle markdown, sorry for the mess.
My understanding is that this means that the uploads can finish without my completion handler from putFile ever running, which is a problem for my app's retry logic.
I guess this is my other question... one of the main decisions we made when originally designing the SDK is that developers should not have to write retry logic; your app will fire the callback when the upload succeeds, fails for authz issues, or times out. Obviously this is complicated if the upload is resumed and your handlers aren't attached (I assume this is the issue you're running into?). Mind posting a code snippet that can be repro'ed?
As a side note, I don't think we've ever tested a long running operation that goes over the authorization token boundary. Not sure if Firebase Auth would have to have background URL fetch enabled to get a new token, and I have no idea if the backend is going to be OK with us swapping tokens like that (my first take is that I don't see why not, since I'm pretty sure the BE doesn't need the user token for anything internally), but it's something that at least would have to be tested before this functionality was rolled out.
Well the issue for me is that if the upload "times out", that usually means "it will work if you try again". My retry logic is present because the upload MUST eventually succeed, so it catches all the errors, decides which ones are retryable, and retries until success.
You're correct that the issue I ran into was that my app tried to upload ABC and failed due to no internet, then later my app started, GTMSessionFetcher immediately rehydrated from a background URLSession and started uploading file ABC, and then my retry logic kicked in and said, "Hmm, looks like ABC never got uploaded, let's upload that now" -- so the file was uploaded twice (not sure if concurrently; I guess so). It's not clear to me why GTMSessionFetch rehydrated it but i'm almost certain it did that from the background URLSession (via identifier), which is why I disabled that functionality.
My retry logic is complicated / large (NSOperation framework) but i'm happy to provide relevant parts if you want me to show you something specific?
Also interested in this for uploading large videos, but don't have the bandwidth right now to roll my own workaround.
I'm also missing the ability to perform large uploads that continue in the background, even when the app is closed, both in android and ios. Writing this comment to raise awareness :)
Any update on this?
Any update?
Any updates here?
@WillBishop Sorry, no concrete plans here yet - other than we'll take another look as we do more investigation on leveraging Swift Concurrency in the implementation.
Well it's 3 years after I was commenting on this issue originally, and 6 years after the issue was filed. I don't see any code changes from Google to fix the current situation, which is what no one expects:
Six. Years.
So for the last three years, in order to stop my framework from being broken by sudden background session rehydrations on app startup, every time I have had to adopt a new release of this SDK I have had to manually patch GTMSessionPatcher after forking the entire Firebase iOS SDK. Now from v9 -> v10, GTMSessionFetcher has gone up a major revision. I want this to just get fixed.
So I am once again asking (Bernie Sanders style) Google please can you either:
GTMSessionFetcher
exposes for you to do this, like setUseBackgroundSession
, or,@xaphod Sorry for the slow progress on this feature and thanks for restating.
To clarify, it sounds like your requirement (1) is the opposite of this feature request's title(2)? You would like to forbid background uploads and downloads instead of enabling them.
And from your draft PR, it sounds like it's sufficient to add a property to the Storage API that sets the right flag for GTMSessionFetcher?
And because background sessions, the complexity of reattaching progress listeners discussed at https://github.com/firebase/firebase-ios-sdk/issues/147#issuecomment-318193709 is not necessary.
Hi @paulb777 no worries, I'm frustrated at the corporation not the people 😄
Yes you are correct, for me personally I want the ability to ensure there are never any background sessions. This is the opposite of what I expect most people want - most people want the ability to actually use a background session properly - so I don't expect "just my needs" to get met here.
I'm hopeful that when this area is considered as a whole, both options will become possible with the Storage API: either to use background sessions properly, or to make sure background sessions are never used at all. IMO the latter should be the default, as I think most people do not understand that uploads are automatically capable of completing in the background but without the completionHandlers being called (ever).
Hi @xaphod - Taking a deeper looks in the debugger, I'm not able to reproduce your issue (or fix). The default value for useBackgroundSession
is NO
, so doing [fetcher setUseBackgroundSession:NO];
is a no-op.
I do see self.usingBackgroundSession = YES;
executing as you describe, but useBackgroundSession
remains NO
.
When an UploadTask/DownloadTask is created when the app is in background, it does leverage NSURLSession's background transfer and "discretionary" is set to true by default by the OS:
https://developer.apple.com/documentation/foundation/nsurlsessionconfiguration/1411552-discretionary
The result is that the task will be performed at the OS's discretion, like only if the device is WiFi connected.
However, the current API is limited in two aspect:
Sometimes an app would like the task to proceed in background no matter what (with "discretionary" set to false). There's no way in Firebase API to specify this.
Usually an app would create the UploadTask/DownloadTask when it is running in foreground, but expect the task to continue running in background and at the OS's discretion (i.e. NSURLSession created with [NSURLSessionConfiguration backgroundSessionConfigurationWithIdentifier] and with "discretionary" set to true). This is useful for performing large files transfer without requiring the user to keep the app opened and simultaneously give discretion to the OS to help avoid consuming precious cellular bandwidth.
Hence, it'll be best if Firebase iOS can provide explicit API for an app to specify if OS discretion is allowed in an UploadTask/DownloadTask, and that the task can be run in background (even if the task is created when the app is in foreground).