App throws ForegroundServiceStartNotAllowedException when in background (Android 12+)

[zahichemaly]

I am testing using the latest BeaconReferenceApplication, with foreground service being enabled.

Expected behavior

Foreground service should restart after we stop monitoring for a while.

Actual behavior

App throws ForegroundServiceStartNotAllowedException when in the background, and foreground service fails to restart.

Logs:

2022-04-15 16:22:50.628 32675-32675/org.altbeacon.beaconreference I/BeaconManager: BeaconManager started up on pid 32675 named 'org.altbeacon.beaconreference' for application package 'org.altbeacon.beaconreference'. isMainProcess=true 2022-04-15 16:22:50.635 32675-32675/org.altbeacon.beaconreference D/BeaconParser: Parsing beacon layout: m:2-3=beac,i:4-19,i:20-21,i:22-23,p:24-24,d:25-25 2022-04-15 16:22:50.639 32675-32675/org.altbeacon.beaconreference D/BeaconParser: Parsing beacon layout: m:2-3=0215,i:4-19,i:20-21,i:22-23,p:24-24 2022-04-15 16:22:50.648 32675-32675/org.altbeacon.beaconreference D/CompatibilityChangeReporter: Compat change id reported: 160794467; UID 10555; state: ENABLED 2022-04-15 16:22:50.652 32675-32675/org.altbeacon.beaconreference W/BeaconManager: Disabling ScanJobs on Android 8+ may disable delivery of beacon callbacks in the background unless a foreground service is active. 2022-04-15 16:22:50.656 32675-32675/org.altbeacon.beaconreference I/ScanJob: Using immediateScanJobId from manifest: 208352939 2022-04-15 16:22:50.660 32675-32675/org.altbeacon.beaconreference I/ScanJob: Using periodicScanJobId from manifest: 208352940 2022-04-15 16:22:50.660 32675-32675/org.altbeacon.beaconreference W/BeaconManager: Disabling ScanJobs on Android 8+ may disable delivery of beacon callbacks in the background unless a foreground service is active. 2022-04-15 16:22:50.661 32675-32675/org.altbeacon.beaconreference I/ScanJob: Using immediateScanJobId from manifest: 208352939 2022-04-15 16:22:50.664 32675-32675/org.altbeacon.beaconreference I/ScanJob: Using periodicScanJobId from manifest: 208352940 2022-04-15 16:22:50.667 32675-32675/org.altbeacon.beaconreference W/BeaconManager: Setting a short backgroundBetweenScanPeriod has no effect on Android 8+, which is limited to scanning every ~15 minutes 2022-04-15 16:22:50.669 32675-32675/org.altbeacon.beaconreference I/BackgroundPowerSaver: We have inferred by application.onCreate in the call stack that we are in the background. 2022-04-15 16:22:50.671 32675-32675/org.altbeacon.beaconreference D/BeaconManager: updating background flag to true 2022-04-15 16:22:50.671 32675-32675/org.altbeacon.beaconreference D/BeaconManager: updating scan period to 1100, 0 2022-04-15 16:22:50.671 32675-32675/org.altbeacon.beaconreference D/BeaconManager: This consumer is not bound. Binding now: org.altbeacon.beacon.BeaconManager$1@5ac25d1 2022-04-15 16:22:50.671 32675-32675/org.altbeacon.beaconreference D/BeaconManager: Binding to service 2022-04-15 16:22:50.671 32675-32675/org.altbeacon.beaconreference I/BeaconManager: Starting foreground beacon scanning service. 2022-04-15 16:22:50.672 1338-1741/? W/ActivityManager: Background started FGS: Disallowed [callingPackage: org.altbeacon.beaconreference; callingUid: 10555; uidState: CEM ; intent: Intent { cmp=org.altbeacon.beaconreference/org.altbeacon.beacon.service.BeaconService }; code:DENIED; tempAllowListReason:; targetSdkVersion:32; callerTargetSdkVersion:32; startForegroundCount:0; bindFromPackage:null] 2022-04-15 16:22:50.673 1338-1741/? W/ActivityManager: startForegroundService() not allowed due to mAllowStartForeground false: service org.altbeacon.beaconreference/org.altbeacon.beacon.service.BeaconService 2022-04-15 16:22:50.678 32675-32675/org.altbeacon.beaconreference D/AndroidRuntime: Shutting down VM 2022-04-15 16:22:50.686 32675-32675/org.altbeacon.beaconreference E/AndroidRuntime: FATAL EXCEPTION: main Process: org.altbeacon.beaconreference, PID: 32675 java.lang.RuntimeException: Unable to create application org.altbeacon.beaconreference.BeaconReferenceApplication: android.app.ForegroundServiceStartNotAllowedException: startForegroundService() not allowed due to mAllowStartForeground false: service org.altbeacon.beaconreference/org.altbeacon.beacon.service.BeaconService at android.app.ActivityThread.handleBindApplication(ActivityThread.java:7463) at android.app.ActivityThread.access$1700(ActivityThread.java:310) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2277) at android.os.Handler.dispatchMessage(Handler.java:106) at android.os.Looper.loopOnce(Looper.java:226) at android.os.Looper.loop(Looper.java:313) at android.app.ActivityThread.main(ActivityThread.java:8611) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:563) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1133) Caused by: android.app.ForegroundServiceStartNotAllowedException: startForegroundService() not allowed due to mAllowStartForeground false: service org.altbeacon.beaconreference/org.altbeacon.beacon.service.BeaconService at android.app.ForegroundServiceStartNotAllowedException$1.createFromParcel(ForegroundServiceStartNotAllowedException.java:54) at android.app.ForegroundServiceStartNotAllowedException$1.createFromParcel(ForegroundServiceStartNotAllowedException.java:50) at android.os.Parcel.readParcelable(Parcel.java:3345) at android.os.Parcel.createExceptionOrNull(Parcel.java:2432) at android.os.Parcel.createException(Parcel.java:2421) at android.os.Parcel.readException(Parcel.java:2404) at android.os.Parcel.readException(Parcel.java:2346) at android.app.IActivityManager$Stub$Proxy.startService(IActivityManager.java:6897) at android.app.ContextImpl.startServiceCommon(ContextImpl.java:1926) at android.app.ContextImpl.startForegroundService(ContextImpl.java:1892) at android.content.ContextWrapper.startForegroundService(ContextWrapper.java:796) at org.altbeacon.beacon.BeaconManager.bindInternal(BeaconManager.java:438) at org.altbeacon.beacon.BeaconManager.autoBind(BeaconManager.java:1810) at org.altbeacon.beacon.BeaconManager.startMonitoring(BeaconManager.java:1136) at org.altbeacon.beaconreference.BeaconReferenceApplication.onCreate(BeaconReferenceApplication.kt:78) at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1211) at android.app.ActivityThread.handleBindApplication(ActivityThread.java:7458) at android.app.ActivityThread.access$1700(ActivityThread.java:310)  at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2277)  at android.os.Handler.dispatchMessage(Handler.java:106)  at android.os.Looper.loopOnce(Looper.java:226)  at android.os.Looper.loop(Looper.java:313)  at android.app.ActivityThread.main(ActivityThread.java:8611)  at java.lang.reflect.Method.invoke(Native Method)  at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:563)  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1133) 

Steps to reproduce this behavior

1. Use targetSdkVersion and compileSdkVersion 31 or above
2. Start the beacon app, then stop ranging and monitoring so the foreground service is stopped
3. Close the app
4. App will crash in the background after a while

Mobile device model and OS version

Galaxy S21 FE 5G (SM-G990E) Android 12

Things I noticed:

• Android 12+ versions have new restrictions regarding foreground service being started in the background, based on this documention, and can only be started in some exceptional cases.
• Foreground service will auto-start after a while. Can we disable this behavior somewhere in the library? I tried disabling the foreground service scanning altogether after stopping the monitoring using disableForegroundServiceScanning, but the service keeps restarting.
• Issue disappears when setting the battery restriction of the app to "Unrestricted", the default being "Optimized"

[davidgyoung]

I believe your analysis of the issue is correct, @zahichemaly. Unfortunately, Android's evolving rules about foreground services don't make this easy.

Whether or not the library starts up with a foreground service is configurable. A few strategies with the exiting library:

1. Many apps that don't need a foreground service don't ever configure one. So this is not an issue in those cases.
2. Some apps start up without a foreground service, and switch to using one only when the app is brought to the foreground (e.g. from an Activity). This works fine on all Android versions and there is no crash.
3. Some apps ALWAYS start up with a foreground service. This works fine, but only on Android 4-11.x.
4. For apps that always start up with a foreground service on Android 12, they will crash with the exception you provided on a background launch. A background launch will happen on beacon detection if you had previously exited the app while scanning for beacons. You can ignore this crash. If you don't want the crash to happen, you can shut down beacon ranging and monitoring before exiting the app.5.
5. Another common option for greater control over the foreground service is to not use the library's foreground service at all and use one you build yourself. Then tell the library to use a regular background service for scanning by beaconManager.setEnableScheduledScanJobs(false). This will use a regular background service for scanning subject to background rules, but if you app also has another foreground service, that scanning service will get all the benefits of the foreground service.6.
6. You may also want to consider using the new IntentScanStrategy documented here https://github.com/AltBeacon/android-beacon-library/pull/1030

I realize all this is complex. I am open to ideas on improving the library to make its behavior more intuitive, but I'm not sure what that would be. We could change the library to refuse to start up the foreground service (if so configured) until the app is in the foreground. But I worry this change would be unexpected and surprising to novice users and may cause unintended side effects. I am not sure this is better than letting the exception happen, and four months from now Android might change the rules all over again!

[zahichemaly]

Thanks @davidgyoung for the detailed explanation. I will look into those alternatives. Regarding the foreground service scanning on Android12+, the user is currently aware that the app is crashing through an "app not responding" popup, and also in the app manager which details the issue. The only workaround so far is changing the battery setting of the app to "Unrestricted", which is not intuitive. So for the meantime, I suggest to catch the exception being thrown and log it in the library just to improve the user experience. This of course only applies to Android 12+. What do you think?

[davidgyoung]

I am a little worried that catching the exception will mask the problem and hide it from users. You found this problem because you saw the exception, right?

Here's an alternative proposal:

1. Deprecate the method beaconManager.enableForegroundServiceScanning and add a deprecation reason explaining the Android 12 problem. The deprecation make make this more visible.
2. Create a new replacement non-deprecated method named something like beaconManager.enableForegroundServiceScanningWhenPossible. It would behave just like the existing method, except if a foreground service could not be started on Android 12, it would log a warning and fall back to using the default job scheduler mechanism of scheduling scans. The restrictions on foreground services would would be documented in the method.
3. We could also change the deprecated method to behave exactly like the new method (2). This would make apps still work to scan beacons (within the limitations of the job scheduler) but would protect against the crashes. I am uncertain if this is a good idea because it still masks a lesser problem (the foreground service fails and scanning is degraded).

[davidgyoung]

One other idea…

if when preparing to start the foreground service, the library could check three conditions:

1. If the library is in background mode
2. If the target SDK is 31+
3. If we are running on Android 12+

If all three of the above are true, the app could refuse to start the foreground service to prevent a crash. Instead it could:

1. Start an intent-backed background scan like is done when using scan jobs during the between scan period.
2. Set a flag indicating a foreground service is desired upon detection.

If a detection comes in via intent the flag above could be checked and the foreground service started at that time.

[davidgyoung]

I am experimenting with an implementation similar to my last comment, but using ScanJob as the fallback strategy. So it would work this way:

1. Try to start a foreground service to scan as commanded by the user.
2. If the above fails: (a) catch the exception (b) log a warning (c) note the user wants to have a foreground service (c) fall back to using ScanJob to scan
3. If and when a bluetooth detection is delivered via intent (this happens with the library when beacons are around and you are between ScanJobs) then we are now allowed to start a foreground service. If we have noted the user wants to do so, try to start at this time.

[davidgyoung]

My first attempt at this is not working yet. Not sure why:

05-22 19:09:15.811 14753 14753 D StartupBroadcastReceiver: onReceive called in startup broadcast receiver
05-22 19:09:15.816 14753 14753 D StartupBroadcastReceiver: Passive background scan callback type: 1
05-22 19:09:15.816 14753 14753 D StartupBroadcastReceiver: got Android background scan via intent
05-22 19:09:15.816 14753 14753 I BeaconManager: unbinding all consumers for failover from intent strategy
05-22 19:09:15.817 14753 14753 D BeaconManager: Unbinding
05-22 19:09:15.817 14753 14753 D BeaconManager: Not unbinding from scanning service as we are using scan jobs.
05-22 19:09:15.817 14753 14753 D BeaconManager: Before unbind, consumer count is 1
05-22 19:09:15.818 14753 14753 D BeaconManager: After unbind, consumer count is 0
05-22 19:09:15.818 14753 14753 I BeaconManager: Cancelling scheduled jobs after unbind of last consumer.
05-22 19:09:15.822 14753 14753 I ScanJob : Using immediateScanJobId from manifest: 208352939
05-22 19:09:15.824 14753 14753 I ScanJob : Using periodicScanJobId from manifest: 208352940
05-22 19:09:15.826 14753 14753 I BeaconManager: binding all consumers for failover from intent strategy
05-22 19:09:15.827 14753 14753 D BeaconManager: Need to rebind for switch to foreground service
05-22 19:09:15.827 14753 14753 D BeaconManager: Binding to service
05-22 19:09:15.827 14753 14753 I BeaconManager: Attempting to starting foreground beacon scanning service.
05-22 19:09:15.834 14753 14753 W BeaconManager: Foreground service blocked by Android Security Exception.  Falling back to job scheduler

[davidgyoung]

Based on my test above, it appears that getting bluetooth scan results delivered by Intent is not sufficient to allow an app to start a foreground service.

The documentation here suggests that starting a foreground service should be allowed in the case that:

Your app receives a Bluetooth broadcast that requires the BLUETOOTH_CONNECT or BLUETOOTH_SCAN permissions

Tests show that delivery of BLUETOOTH_SCAN Intents do not cause this to be allowed. I dug through the AOSP source code, and it seems that bluetooth connection events are rally the only events that do cause this to be allowed.

This means that it is impossible for the library to auto-restart the foreground service at a later time (unless the app comes to the foreground), because the library itself handles no background events that make it eligible. As a result, the best the library could do is:

1. Automatically fall back to using ScanJob if a foreground service cannot be started, thus avoiding a crash and maintaining as much functionality as possible.
2. Provide a new API to see if the above has happened.
3. Provide a new API to retry starting the foreground service if the app has reason to believe an event has happened that would allow it.
4. If the library detects that the app does come to the foreground, automatically try to start the foreground service.

[davidgyoung]

This is fixed in #1088

[davidgyoung]

As of #1088 and the latest beta version of this library, if you configure a foreground service, the library will try to use it, but if Android 12 blocks it, it caches the Exception and falls back to using the Job Scheduler. If the app moves to the foreground, the library will automatically switch back to using a foreground service (because it is allowed to do so at that time.)

I also added a way to check if the foreground service start failed, and retry starting it if the app knows that a motion, geofence or other event happened that would temporarily cause the operating system to allow a foreground service start.

Details on how this works are here: https://altbeacon.github.io/android-beacon-library/foreground-service.html