Figure out how to deal with Doze (Android 6+)

[cketti]

Doze reduces battery consumption by deferring background CPU and network activity for apps when the device is unused for long periods of time.

Learn more about it here: Optimizing for Doze and App Standby

As far as I can tell we have these options:

1. Embrace Doze. When the device tells us to go to sleep we do.
2. Fight Doze. Use the new API to schedule checks for new mail (can't be more frequent than every 15 minutes)
3. Evade Doze. Foreground services (think music player) prevent the device from entering Doze. This requires displaying an ongoing notification.

As far as I can tell users can't disable Doze in settings. They have, however, the option to put apps on a whitelist.

An app that is whitelisted can use the network and hold partial wake locks during Doze and App Standby. However, other restrictions still apply to the whitelisted app, just as they do to other apps. For example, the whitelisted app’s jobs and syncs are deferred, and its regular AlarmManager alarms do not fire.

So the interval between mail checks still has to be at least 15 minutes. But there is a chance that Push might work during Doze. Keeping a connection open doesn't require alarms.

My goal is to first collect all options we have and then decide on the best way forward. Please let me know if you can think of any other options and I will add them to this post. Also, if you have a special use-case regarding mail checks that you think should be supported by K-9 Mail let us know.

Update (2015-11-10): It looks like the only way for an app to prevent the device from entering Doze is to keep the screen on. So option 3 is not something that's feasible for us.

Update (2015-11-20): Using a foreground service is possible but tricky due to a bug in Android. See https://commonsware.com/blog/2015/11/18/another-doze-edge-case-foreground.html

[sidamos]

High priority GCM notifications do work normally during Doze (that does not help K9, of course), so Hangout messages do work. For me, mails are as important as Hangout messages. So, doing nothing about Doze (like Gmail does not fetch IMAP) is not on option for me.

I think, push should work if the user whitelists the app (this setting allegedly helps WhatsApp). If the user enables push, K9 could ask the user to put the app on the whitelist (there is an API for that). I have not tested that, because I do not use push.

For periodic mail checking, K9 should use the new AlarmManager methods. K9 should have an option to do that. This still allows the device entering Doze mode and saves battery. I would not call that "fighting Doze".

Having an ongoing notification is annoying IMHO and a mail app should not prevent the device from going into Doze at all.

[ssieb]

If whitelisting the app is required to make push and/or poll to work, then I'm fine with that. As mentioned above, there should be some notification to the user that it's necessary to do so.

[NadirHusain]

Does anyone know for sure how much " inactive " time needs to elapse before doze mode kicks in ?

[cketti]

@NadirHusain: As far as I can tell OEMs are free to tweak the different timeout values. See DeviceIdleController. From what I've read current Nexus devices enter Doze after 1 hour without significant motion.

[NadirHusain]

I agree with @sidamos . Doing nothing is not an option. Since I am a push user ( and perhaps driven by that) I feel push should be a top priority. Alarm manager should be used to fetch . Sorry I can't code so really can't contribute in any tangible way except just to put forth my opinion. Thanks

[gdt]

Doze mode seems like a reasonable concept, except that it's trying to push people into GCM (which is non-Free and has privacy issues). I would suggest going down two paths simultaneously

• Try to be a good doze client. With polling, just allow the polling interval to get longer as doze requires it. That might be as simple as setting an alarm for now+poll_interval on every wakeup and not worrying that it's longer than configured. This is not really so crazy, as the idea that one needs rapid polling when the device is really idle only seems to fit a use case where one leaves the phone on a table in a room and stays there for hours. I guess it will be interesting to see logs of real people and if their phones doze when they are not asleep or really away from them. Also with this, intercept some done_dozing signal and poll again if it's been longer than the configure interval. Perhaps also set some doze-mode poll interval, so the user can say poll every 10 minutes, but only every hour in doze mode, if that's what they want. Or maybe that's too complicated.
• Make push still work. It seems we really don't understand enough yet, and whitelisting is the only way. Somewhat eparately from this, it would be nice if push could have much longer times in IDLE, to make k-9 use less power in doze mode.

What I don't see being all that useful or important (to me) is enabling polling in doze more often than every 15 minutes. I suppose that could be important to people who can't use push, need to get mail promptly, and leave their phone on a table while being near it. It would be interesting to hear from anybody who sees themselves in that situation.

For me, if my phone goes into doze because I haven't touched it and it hasn't moved in over an hour, having mail arrive up to 15 minutes late is ok. Unless I am mis-guessing how often doze will happen.

[sidamos]

Periodic check: You don't have to implement, what you are describing in your 1st paragraph, Android does that already. If you use the old AlarmManager methods, they won't fire during Doze, unless the device enters the maintenance window (every hour or more). If you use the new AlarmManager methods, alarms are being throttled to fire every 15 minutes at most. My use case is this: During work, the phone lies on my table for longer periods of time. But I don't want to get mails only every hour or even later. Also, Doze is supposed to end, when you move the device. But in my test, it only ends if I switch the display on. And I don't want to switch the display on every now and then only to enable K9 to look for new mails. So, K9 should use the new AlarmManager method, if the user desires to get mails during Doze.

Push: If we are lucky, push should work as normal if K9 is on the whitelist. So, nothing has to be implemented for this. Optionally, K9 could ask the user to be put on the whitelist. There is an intent for that.

[NadirHusain]

As an aside I would like to mention that I only get real push if my " refresh idle connection" is set to 6 minutes or less. Any higher number and my messages are delayed. I understand my cellular operator is most likely responsible for this but was just wondering if this option would still be available to me in the doze mode ? I am sure there are others like me and k9 needs to be able to " force " this incase doze " kills " this option in default mode . Thanks

[cketti]

My current plan is:

1. By default we embrace Doze. If Push is enabled we disable it while Doze is active. That will prevent the app from establishing a Push connection during the Doze maintenance windows.
2. If K-9 Mail was whitelisted by the user we use the new API for our alarms. That should allow poll frequencies of up to every 15 minutes and Push should work, too. The Push refresh interval can't be smaller than 15 minutes though.

[NadirHusain]

as of now K9 5.107 android 5.1.1 i need a minimum refresh time of 6 minutes to be able to get real time push notifications (tcp/cellular service provider issue). setting the minimum idle refresh time at 15 minutes would mean end of push mail for me (as i know it) unless there is a way to set idle refresh time at 6 minutes or less when doze mode is NOT active and could switch to 15 minutes when doze kicks in. i am sure there are many others like me who need real time push and have idle refresh set at very low intervals . request you to consider this use case issue . thanks

[cketti]

@NadirHusain: Doze is only available on Android 6+. If Doze is not active (or on previous Android versions) K-9 Mail will not impose any new restrictions.

[NadirHusain]

@cketti clear now . thanks .

[sidamos]

Good idea. This should have been done by Google, without the need to call new AlarmManager methods by the app.

[imsaguy]

Any progress on this? If I have to look at K9 periodically just to get my notifications, I might as well set my sync to manual which defeats the whole purpose of push notifications. If you're looking at ways to bypass, do what apps like KeepPass do and set a transparent notification icon which keeps the app in the foreground like a music app and prevents doze.

[daradib]

@imsaguy For now you can manually whitelist K9 under Settings > Battery > Battery Optimization (or similar). Mostly fixes push and #970. Doze still partially applies, since current AlarmManager will only fire during increasingly spaced apart maintenance windows, so I think the IMAP IDLE connection may eventually time out if there is no traffic and the device is stationary long enough. Also, to be specific, the foreground service prevents App Standby but not Doze.

[gadgetfan10]

@cketti So under the "embrace Doze" plan, is the idea to stop push connections during Doze but - and this would seem to be the difference from the user end - re-enabling push connections when the device comes out of Doze?

Sort of separately, and maybe because I don't understand the specifics, why would this approach prevent the establishment of push connections during the Doze maintenance windows?

[cketti]

@gadgetfan10 Poor wording on my part.

Embrace Doze:

• Close existing Push connections when entering Doze; don't attempt to create new ones while in Doze or maintenance window
• During maintenance windows we trigger a simple mail synchronization
• Return to normal operation (e.g. establish Push connections) when exiting Doze (device returns to normal operation; not just Doze maintenance window)

[gadgetfan10]

Sounds like a solid plan. Thanks for the clarification.

Looking forward to seeing it implemented. If I had programming skills, I'd offer to help. Since I don't, I'll just say thanks for your efforts.

[doodhout]

@cketti As I understand how this will work: is there no way to enable doze and keep push connections alive during screen-off? Right now, I'm using a Sony smartphone that features similar functionality called 'Stamina Mode' and by whitelisting K9Mail I am ensured the K9Mail app works as if Stamina Mode hasn't been enabled - other apps that are not whitelisted will still be forced to go to sleep and ceize their activity.

Doze seems to be more refined than Stamina Mode, as it offers an API that allows apps certain activity but still well controlled and more battery-friendly than if the feature was disabled altogether; which isn't the case for Stamina Mode.

[bobpullen]

Just a note to say that this issue is exacerbated on Android N where Doze 2.0 is initiated on screen off rather than a perceived period of inactivity.

[chrcoluk]

is there a proper workaround to this?

I can confirm poll checks are not working, I just whitelisted k9 in the power saving options, and am waiting to now see if the poll works.

[doodhout]

@chrcoluk Even when whitelisted, the poll checks will only occur in (but not outside of) the windows of activity that Doze permits, and the time between those windows will increase continuously thus delaying poll checks by minutes or hours!

[chrcoluk]

yeah so this is now broken for the forseeable future then?

I thought I managed to disable doze on the entire phone by turning of the battery savings feature, but seems it is still in use. :(

The phone is samsung touchwiz 6.0.1.

[chrcoluk]

According to greenify dev if you add GCM priority option to the app and the user enables it, then the app can bypass doze mode.

[garfieldthecat]

@chrcoluk Not sure I understand the comment about GCM priority; I mean, I understand that an app sending high priority GCM notifications will cause the phone to exit Doze mode, but k9 doesn't use GCM, so how is this helpful?

[garfieldthecat]

I have found this partial solution works for me: http://androidforums.com/threads/marshmallow%E2%80%99s-doze-broke-push-notifications-my-fix-is-preventing-doze-with-macrodroid.1058445/ I use MacroDroid to 'do something' at regular intervals, preventing the phone from entering Doze. The battery usage is OK. It does not solve the issue of getting push with Doze, but it does - indirectly - solve the issue of getting push email on an unrooted Marshmallow device. The post also has details on Aquamail and GCM-based clients; in summary, as of now there is no solution to get proper push email with Doze, because no email app sends high-priority GCM notifications. Google has gone crazy.

[garfieldthecat]

Has any one figured out with certainty if push (imap idle) never works when in Doze, or only works under certain conditions?

[chrcoluk]

I ended up disabling doze.

on XDA is a app called "doze settings editor" Needs root, you can fine tune all the doze variables, this app is unique in that it allows one to tone doze down rather than make it more aggressive that all the other doze related apps seem to do.

I set the interactive timeout to 86400secs (one day) so basically the phone now needs one day to enter doze mode, effectively disabling it for me as I use the phone daily. Now K9 works a treat again.

Of course only people with rooted phones and the confidence to adjust this sort of thing can use my fix, I think k9 needs to somehow find a more widespread fix to the problem or it will disappear into irrelevance when doze mode is standard on all devices.

[doodhout]

@chrcoluk: Couldn't you just have whitelisted K9Mail in the Battery Saving menu on your phone? That would effectively disable doze for that app. I've done so and it seems to work... I think.

[bobpullen]

@doodhout whitelisting an app does not completely exempt it from doze - https://developer.android.com/training/monitoring-device-state/doze-standby.html#support_for_other_use_cases

[philipwhiuk]

I think we need to add Intent handling for ACTION_DEVICE_IDLE_MODE_CHANGED to BootReceiver and look at the values of:

• isIgnoringBatteryOptimizations
• isDeviceIdleMode

When we come out of idle mode we need to restart sync if it's been stopped by going into Doze.

When we go in we should stop sync unless isIgnoringBatteryOptimizations is true I expect.

We may also want to look at isPowerSaveMode

See: PowerManager

[philipwhiuk]

Listening for these intents is proving difficult. I tried both Manifest changes and, following advice on StackOverflow, code, but neither seemed to work for my device.

[Valodim]

I heard conversations is doing a good job figuring out the state and informing the user about this, might be worth taking a look at their code

[philipwhiuk]

They request whitelisting:

<uses-permission android:name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS" />

[k3a]

If this can't be solved on app side / whitelisting, please update FAQ (https://k9mail.github.io/documentation/faq.html#anchor15) to inform users what to do as the FAQ doesn't mention that at all ATM.

I have MIUI 8.1.1.0 (based on Android 6.0.1) and it happened to me as well. I think Settings->Battery & performance->Manage apps battery usage->Choose apps->k9 Mail and setting "No restrictions" under "Background settings" fixes it but it probably resets after the K9 is updated, I am not sure. I need more time to watch it if it will work again now after this manual re-whitelisting.

[djdookie]

Because you closed my android 7.1 thread and marked as duplicate of this one. I saw this one and wasn't sure if it is really a duplicate because interestingly I didn't have those problems with CM13 (android 6). Just now with CM 14 (android 7).

[philipwhiuk]

Doze is a lot more aggressive on Android 7 so you're much more likely to run into it. Specifically it no longer matters if the phone moves - it can still doze.

[Bubu]

@djdookie CyanogenMod removed the doze functionality from their Android 6/CM13 builds. It is enabled in CM14 (Android 7) though.

[sidamos]

The implementation of K9 5.2 RC regarding enabling using the new AlarmManager API to be able to fetch mails despite active Doze mode is not optimal. It relies on disabling (whitelisting) battery optimization (standby mode) for K9. But that itself has nothing to do with Doze from an Android OS standpoint. So, K9 is just reusing that settings screen for something else. Problem is, some Android 6.0 implementations do not have this particular settings screen, for example MIUI. I guess, MIUI does not use that "standby" mode of Android 6.0, because they have their own ways of battery optimization. It however makes use of Doze mode. After fresh install of K9 5.2, battery optimization of K9 is enabled despite android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS. I checked that by opening the battery optimization screen via a Nova Launcher Activity shortcut. Because of that, K9 does not fetch mails at specified intervals during Doze. So, the better thing for K9 would be IMHO to have that setting inside K9 owns preferences screens. It makes more sense.

[philipwhiuk]

Okay, so while investigating an entirely separate issue I found the major thing that had been blocking me working on this. Namely that I couldn't work out why we weren't getting notifications of Doze state change. The reason was annoyingly simple. Our target API level for some reason is still 22.

Anyway, as soon as I bumped it up I got notifications of Doze.

Now for proper behaviour all we have to do is notify the mail service that we have changed state (essentially a connectivity change). This will need a bit of work and some testing but it's no longer as blocked as I thought it was.

My current expectations for this issue are:

1. Make it so that when we resume from Doze we reactive push.

This is the biggest problem with K-9 right now. Doze actively breaks K-9 because it acts like a network connectivity change and only forcing a real network change fixes that. So even once Doze ends, right now, you won't get emails until you toggle/leave your WiFi point / etc.

This is now possible with intent handling stuff I mentioned earlier.

2. Add REQUEST_IGNORE_BATTERY_OPTIMIZATIONS and have some UX dialog stuff that explains why we want it (Doze will cause your email to stop being synced).

[cketti]

I don't believe we have to change the targetSdkVersion to fix this. It might "fix" things as a side-effect because when targetSdkVersion is 24 broadcast receivers declared in the manifest won't receive connectivity changes at all. See https://developer.android.com/about/versions/nougat/android-7.0-changes.html#bg-opt

As an immediate fix we should probably register a broadcast receiver to listen for doze change events and adjust our internal connectivity state accordingly. We also only want to poll in maintenance windows and not establish a new push connection that will be killed moments later anyway.

[philipwhiuk]

As an immediate fix we should probably register a broadcast receiver to listen for doze change events

Having tested it, I don't think you can get Doze change events unless you target v23.

[Valodim]

Bumping minSdk to 21 is probably not an option, and maintaining a JobScheduler implementation next to AlarmManager (which none of the devs will have in daily use then) sounds very painful. We should probably look at https://github.com/evernote/android-job to help with this?

[ryancdotorg]

I had a hard time figuring out how to whitelist k-9 from battery optimizations. For anyone else finding this ticket, on Android 7.1.1 it was:

Settings -> Battery -> Menu (three dots top right of screen) -> Battery optimization -> All apps (switch from "Not optimized" -> K-9 Mail -> Don't optimize

Hope this helps someone. Perhaps this condition should be detected? It seems possible for an app to change its setting, tasker did.

[ryancdotorg]

Whitelisting from battery optimizations doesn't seem to help. :-(

[coogor]

Yes, this was already determined in #2242 From the discussion above, I could not derive a clear recommendation how to get push running again under Android 7. Looks like a coding solution is required?

[Valodim]

Yes. Our current best idea is to rewrite the push code using the android-job library mentioned above. However this is not a simple task, so it's difficult to give a timeline :hurtrealbad:

[chadwickblane]

Seems like an evil google conspiracy to coerce use of gcm. Feed your souldata to the google lake of fire.

[chadwickblane]

I optimize my battery by uninstalling all google apps and all provider apps [verizon]. The battery lasts much longer, and the handset uses less data.

[philipwhiuk]

This isn't a discussion forum @chadwickblane. Please avoid adding stuff that doesn't contribute productively. For what it's worth, many non stock ROMs implement Doze as well.