fix: ANR when using Firebase Cloud Messaging with Shorebird

[eseidel]

Report from a customer:

Definitely looks like our bug.

There are at least two bugs I could imagine here:

1. Launching a second Flutter engine (if it goes though flutter_main) could end up blocked on our global lock for rust config if the background thread is still doing the update download.
2. If it's trying to launch with something other than libapp.so, we might end up launching from the wrong Dart bundle (we should check that they're targeting libapp.so).

In either case, we need to make an example app which uses FlutterFirebaseMessagingBackgroundExecutor and see if we can reproduce: https://github.com/firebase/flutterfire/blob/master/packages/firebase_messaging/firebase_messaging/android/src/main/java/io/flutter/plugins/firebase/messaging/FlutterFirebaseMessagingBackgroundExecutor.java

[bryanoltman]

I set up a test project with FCM and was able to receive and open background notifications without issue. One thing I'm noticing from the crash log is the split_config.arm6... in the left column. Maybe FCM is a red herring here?

[bryanoltman]

Going through the steps of adding crashlytics to our demo app and noticed the bit in https://firebase.google.com/docs/crashlytics/get-started?platform=flutter#add-sdk about "If your Flutter project uses the --split-debug-info flag"

[eseidel]

Several issues here, starting to catalogue them

1. Firebase will end up causing Flutter to call nativeInit twice. We don't expect that, and won't want to init twice.
2. However we will want any second-init to use the current boot patch (and not have to wait on a lock to do so).

[eseidel]

We believe we understand the issue now.

The hang is believed non-fatal (the app doesn't crash, or hang indefinitely), but will hang (in the background) for some period of time thus triggering ANR reports from Android/Crashlytics. We believe it occurs when:

1. FCM triggers in the background.
2. Shorebird is being used in the app.
3. Multiple FCM messages are queued.
4. The network is slow/unreliable.

We have a fix planned and being worked on. It's a slightly involved refactoring of our Rust code which will likely take most of today but should remove this entire class of issues. I expect us to release a fix tomorrow. We also have a (planned) release coming out today, (0.5.1) which will not have this fix in it yet.

Thank you for your patience.

[eseidel]

Fixes planned:

• [ ] 1. Fix rust to only allow initing shorebird once per process (FCM will init Flutter repeatedly, despite Flutter engine warnings). This is where the hang happens in the reported stack trace, but if we fixed this alone it would still hang on the next call because the updater lock is being held by a long network call.
• [ ] 2. Fix rust to not hold the updater lock during long network pauses. This is the real fix to the issue. We don't need to hold the updater lock across a network call, just will require some refactoring to fix.
• [ ] 3. Fix rust code to not allow spawning multiple updater threads (this isn't possible today, but might become possible after we fix holding the lock during the updater thread network calls).
• [x] 4. Fix Engine code to not call into Shorebird when not using libapp.so and log a warning. This is a speculative fix noticed while in this code. Although FCM is still using libapp.so, other callers of Flutter might not, and it appears we will do the wrong thing for them, so might as well fix it while we're there.

[bryanoltman]

Fix Engine code to not call into Shorebird when not using libapp.so and log a warning. This is a speculative fix noticed while in this code. Although FCM is still using libapp.so, other callers of Flutter might not, and it appears we will do the wrong thing for them, so might as well fix it while we're there.

Addressed in https://github.com/shorebirdtech/engine/pull/31

[eseidel]

I'm pleased to report that I have a fix compiling for 2. and 3, testing now.

[linhmanh92]

I am the one who reported this error to you on discord. glad that the fix is still going well. Thank you.