Deadlock in -[APMIdentity appInstanceID]

bryansum commented 1 year ago

Description

We've been investigating some strange behaviors in our app that were are hard to reproduce, but manifested when foregrounding the app and as we fired off requests to refresh our content (and also send analytics events).

Eventually we figured out it seemed like some kind of contention / deadlock issue.

Eventually we were able to catch an instance and attached the debugger, where we were able to see the stack trace. The root cause seems to be a deadlock caused by contention on the dispatch_sync call inside -[APMIdentity appInstanceID] as per this screenshot:

In our case we were using Swift concurrency which has has a number of concurrent system threads matching the number of physical cores on the system (in my case this was 6 with an iPhone 14 Pro) servicing Swift Tasks. Each thread of these 6 was blocked on the same dispatch_sync call and no forward progress could be made.

It would stand to reason that this issue could also occur in other situations as well where other threads are bounded, not just when using Swift concurrency.

My understanding is that using dispatch_sync is tricky to get right, and may lead to deadlocks when accessed recursively, especially when the caller's context is unknown. According to man 3 dispatch_sync this is conceptually implemented using a dispatch_semaphore, which is described as unsafe to use with Swift concurrency in WWDC 2021 "Swift concurrency: Behind the scenes":

Unfortunately it's hard to introspect the GoogleAppMeasurement package to determine what could be the cause of the deadlock, but we also are using AppCheck, Analytics, Auth, Firestore, Functions, Dynamic Links, and RemoteConfig, among others. I could imagine this could be some kind of case where other mutual exclusion techniques elsewhere in Firebase could trigger this scenario.

Reproducing the issue

Reproducing this is tricky. I tried calling Analytics.appInstanceID in a tight loop concurrently (which was what was hanging) and wasn't able to reproduce. It seems likely that there's some other Firebase logic that has a dependency on GoogleAppMeasurement#APMIdentity that might be causing this issue. You can see that FIRAppAttestProvider.state is also blocked on something in that screenshot (unfortunately I don't have that stack available anymore), but they don't look related doing a basic inspection of FIRAppAppAttestProvider logic.

Firebase SDK Version

10.10

Xcode Version

14.3

Installation Method

Swift Package Manager

Firebase Product(s)

Analytics, App Check, Authentication, Crashlytics, Database, DynamicLinks, Firestore, Functions, In-App Messaging, Installations, Remote Config, Storage

Targeted Platforms

iOS

Relevant Log Output

No response

If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet

```json { "pins" : [ { "identity" : "abseil-cpp-binary", "kind" : "remoteSourceControl", "location" : "https://github.com/google/abseil-cpp-binary.git", "state" : { "revision" : "bfc0b6f81adc06ce5121eb23f628473638d67c5c", "version" : "1.2022062300.0" } }, { "identity" : "amplitude-ios", "kind" : "remoteSourceControl", "location" : "https://github.com/amplitude/Amplitude-iOS", "state" : { "revision" : "9e3081af2f3d45931b5404c4276c6c2e21ecd671", "version" : "8.13.0" } }, { "identity" : "analytics-connector-ios", "kind" : "remoteSourceControl", "location" : "https://github.com/amplitude/analytics-connector-ios.git", "state" : { "revision" : "b44981119e2efc0edd1d2e26831c3c4c0e0264e6", "version" : "1.0.0" } }, { "identity" : "appsflyerframework", "kind" : "remoteSourceControl", "location" : "https://github.com/AppsFlyerSDK/AppsFlyerFramework", "state" : { "revision" : "d1349c8fdd18a80a776298e4c5c3526e22feff78", "version" : "6.8.1" } }, { "identity" : "ats-sdk-ios", "kind" : "remoteSourceControl", "location" : "https://github.com/LiveRamp/ats-sdk-ios.git", "state" : { "revision" : "2983fdd06ab861f9286afb2ded727305441f6a67", "version" : "1.4.0" } }, { "identity" : "combine-schedulers", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/combine-schedulers", "state" : { "revision" : "aa3e575929f2bcc5bad012bd2575eae716cbcdf7", "version" : "0.8.0" } }, { "identity" : "firebase-ios-sdk", "kind" : "remoteSourceControl", "location" : "https://github.com/firebase/firebase-ios-sdk.git", "state" : { "revision" : "5034479ca0c4f32f299677fb0ba1fe4d3e3e20b4", "version" : "10.10.0" } }, { "identity" : "googleappmeasurement", "kind" : "remoteSourceControl", "location" : "https://github.com/google/GoogleAppMeasurement.git", "state" : { "revision" : "69f5b8425dd473d2f2475bba7f88f82cd9da0e36", "version" : "10.10.0" } }, { "identity" : "googledatatransport", "kind" : "remoteSourceControl", "location" : "https://github.com/google/GoogleDataTransport.git", "state" : { "revision" : "5056b15c5acbb90cd214fe4d6138bdf5a740e5a8", "version" : "9.2.0" } }, { "identity" : "googleutilities", "kind" : "remoteSourceControl", "location" : "https://github.com/google/GoogleUtilities.git", "state" : { "revision" : "871d43135925cde39ef7421d8723ce47edfdcc39", "version" : "7.11.1" } }, { "identity" : "grdb.swift", "kind" : "remoteSourceControl", "location" : "https://github.com/groue/GRDB.swift.git", "state" : { "revision" : "dd7e7f39e8e4d7a22d258d9809a882f914690b01", "version" : "5.26.1" } }, { "identity" : "grpc-binary", "kind" : "remoteSourceControl", "location" : "https://github.com/google/grpc-binary.git", "state" : { "revision" : "f1b366129d1125be7db83247e003fc333104b569", "version" : "1.50.2" } }, { "identity" : "gtm-session-fetcher", "kind" : "remoteSourceControl", "location" : "https://github.com/google/gtm-session-fetcher.git", "state" : { "revision" : "d415594121c9e8a4f9d79cecee0965cf35e74dbd", "version" : "3.1.1" } }, { "identity" : "gzipswift", "kind" : "remoteSourceControl", "location" : "https://github.com/1024jp/GzipSwift.git", "state" : { "revision" : "7a7f17761c76a932662ab77028a4329f67d645a4", "version" : "5.2.0" } }, { "identity" : "ios-sdk", "kind" : "remoteSourceControl", "location" : "https://github.com/spotify/ios-sdk", "state" : { "revision" : "c12c8630949492439e54b6482219c34bc7d65157", "version" : "1.2.3" } }, { "identity" : "leveldb", "kind" : "remoteSourceControl", "location" : "https://github.com/firebase/leveldb.git", "state" : { "revision" : "0706abcc6b0bd9cedfbb015ba840e4a780b5159b", "version" : "1.22.2" } }, { "identity" : "libwebp-xcode", "kind" : "remoteSourceControl", "location" : "https://github.com/SDWebImage/libwebp-Xcode.git", "state" : { "revision" : "0f3bdb28a1edc5e8e43876d3835d20c601ef331f", "version" : "1.2.3" } }, { "identity" : "lottie-ios", "kind" : "remoteSourceControl", "location" : "https://github.com/airbnb/lottie-ios", "state" : { "revision" : "314537ec697719fa33a9d48210f4d06a463286d3", "version" : "3.4.3" } }, { "identity" : "nanopb", "kind" : "remoteSourceControl", "location" : "https://github.com/firebase/nanopb.git", "state" : { "revision" : "819d0a2173aff699fb8c364b6fb906f7cdb1a692", "version" : "2.30909.0" } }, { "identity" : "nimbus-ios-sdk", "kind" : "remoteSourceControl", "location" : "https://github.com/timehop/nimbus-ios-sdk", "state" : { "revision" : "becc9351d48d51b8739843ca8a59a216e6c91edf", "version" : "2.10.0" } }, { "identity" : "onesignal-xcframework", "kind" : "remoteSourceControl", "location" : "https://github.com/OneSignal/OneSignal-XCFramework", "state" : { "revision" : "b87060d4a77ab6b5e96d6655b2587d3cb05e33e5", "version" : "3.12.3" } }, { "identity" : "persister", "kind" : "remoteSourceControl", "location" : "https://github.com/Lickability/Persister", "state" : { "branch" : "main", "revision" : "556198feaa1b37cfb54328af43dcb2709002972a" } }, { "identity" : "phonenumberkit", "kind" : "remoteSourceControl", "location" : "https://github.com/marmelroy/PhoneNumberKit", "state" : { "revision" : "9b051490589ae4f2fa861287407a43a87be7226c", "version" : "3.5.8" } }, { "identity" : "promises", "kind" : "remoteSourceControl", "location" : "https://github.com/google/promises.git", "state" : { "revision" : "3e4e743631e86c8c70dbc6efdc7beaa6e90fd3bb", "version" : "2.1.1" } }, { "identity" : "sdwebimage", "kind" : "remoteSourceControl", "location" : "https://github.com/SDWebImage/SDWebImage.git", "state" : { "revision" : "4178d12a44691182c2eb79c70281b14cb73f4cbf", "version" : "5.14.3" } }, { "identity" : "sdwebimageswiftui", "kind" : "remoteSourceControl", "location" : "https://github.com/SDWebImage/SDWebImageSwiftUI.git", "state" : { "revision" : "61fefe9c284fd41ddef77d02749e88f00c305196", "version" : "2.2.2" } }, { "identity" : "sdwebimagewebpcoder", "kind" : "remoteSourceControl", "location" : "https://github.com/SDWebImage/SDWebImageWebPCoder.git", "state" : { "revision" : "4416805045174938b11edb69a86e9f7dd0699952", "version" : "0.9.1" } }, { "identity" : "sentry-cocoa", "kind" : "remoteSourceControl", "location" : "https://github.com/getsentry/sentry-cocoa", "state" : { "revision" : "d277532e1c8af813981ba01f591b15bbdd735615", "version" : "8.8.0" } }, { "identity" : "swift-case-paths", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swift-case-paths", "state" : { "revision" : "870133b7b2387df136ad301ec67b2e864b51dda1", "version" : "0.14.0" } }, { "identity" : "swift-clocks", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swift-clocks", "state" : { "revision" : "20b25ca0dd88ebfb9111ec937814ddc5a8880172", "version" : "0.2.0" } }, { "identity" : "swift-collections", "kind" : "remoteSourceControl", "location" : "https://github.com/apple/swift-collections", "state" : { "revision" : "f504716c27d2e5d4144fa4794b12129301d17729", "version" : "1.0.3" } }, { "identity" : "swift-composable-architecture", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swift-composable-architecture.git", "state" : { "revision" : "3e8eee1efe99d06e99426d421733b858b332186b", "version" : "0.52.0" } }, { "identity" : "swift-custom-dump", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swift-custom-dump", "state" : { "revision" : "de8ba65649e7ee317b9daf27dd5eebf34bd4be57", "version" : "0.9.1" } }, { "identity" : "swift-dependencies", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swift-dependencies", "state" : { "revision" : "6bb1034e8a1bfbf46dfb766b6c09b7b17e1cba10", "version" : "0.2.0" } }, { "identity" : "swift-gen", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swift-gen.git", "state" : { "revision" : "5bd20fb662e1ead7ee47df6bb0a15398295f2e06", "version" : "0.4.0" } }, { "identity" : "swift-identified-collections", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swift-identified-collections", "state" : { "revision" : "ad3932d28c2e0a009a0167089619526709ef6497", "version" : "0.7.0" } }, { "identity" : "swift-nonempty", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swift-nonempty.git", "state" : { "revision" : "8074e5f2e4a6de5a42476e545cc33438021aa2a4", "version" : "0.4.0" } }, { "identity" : "swift-overture", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swift-overture", "state" : { "revision" : "7977acd7597f413717058acc1e080731249a1d7e", "version" : "0.5.0" } }, { "identity" : "swift-package-manager-google-mobile-ads", "kind" : "remoteSourceControl", "location" : "https://github.com/googleads/swift-package-manager-google-mobile-ads.git", "state" : { "revision" : "3a03edf8728c50bc380b4a9187adf178ea477a6d", "version" : "9.12.0" } }, { "identity" : "swift-package-manager-google-user-messaging-platform", "kind" : "remoteSourceControl", "location" : "https://github.com/googleads/swift-package-manager-google-user-messaging-platform.git", "state" : { "revision" : "3b924ce3313a5fd2fc6a8dc889a8c38f76890fe3", "version" : "2.0.0" } }, { "identity" : "swift-protobuf", "kind" : "remoteSourceControl", "location" : "https://github.com/apple/swift-protobuf.git", "state" : { "revision" : "e1499bc69b9040b29184f7f2996f7bab467c1639", "version" : "1.19.0" } }, { "identity" : "swift-tagged", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swift-tagged", "state" : { "revision" : "f3f773a5e13f3c8f0ab1ce2ae6378058acefa87d", "version" : "0.7.0" } }, { "identity" : "swiftui-navigation", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/swiftui-navigation", "state" : { "revision" : "0a0e1b321d70ee6a464ecfe6b0136d9eff77ebfc", "version" : "0.7.0" } }, { "identity" : "vungleadssdk-swiftpackagemanager", "kind" : "remoteSourceControl", "location" : "https://github.com/Vungle/VungleAdsSDK-SwiftPackageManager.git", "state" : { "revision" : "3ffeb9316a625e9486f850da15dff3713dd6f9b5", "version" : "7.0.1" } }, { "identity" : "xctest-dynamic-overlay", "kind" : "remoteSourceControl", "location" : "https://github.com/pointfreeco/xctest-dynamic-overlay.git", "state" : { "revision" : "62041e6016a30f56952f5d7d3f12a3fd7029e1cd", "version" : "0.8.3" } } ], "version" : 2 } ```

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet

```yml Replace this line with the contents of your Podfile.lock! ```

google-oss-bot commented 1 year ago

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

paulb777 commented 1 year ago

Sorry about the trouble and thanks for sharing the thorough analysis. This may be another reincarnation of an issue that has been fixed before. See #10080

bryansum commented 1 year ago

Apropos this discussion, these Swift forum posts talking about deadlocks using DispatchQueue.sync + async barriers / reader-writer locks:

pcfba commented 1 year ago

Thanks for posting the details and links. We're looking into reproducing this and potential solutions.

xidipeng commented 10 months ago

@pcfba @tsunghung the same deadlock in mostly crash in 16.6.x iOS issue_470ed3efa2c2edd015db8803bd488d34_crash_session_886363525a5c4b02a644f615a20b32ea_DNE_0_v2_stacktrace.txt

tsunghung commented 9 months ago

The root blocker is [ASIdentifierManager advertisingIdentifier]. Unfortunately, we haven't heard back from Apple. We added a timeout in dispatch_block_wait as a workaround. It seems that the timeout is not working properly, either.

APMIdentityWorkerQueue
0  libsystem_kernel.dylib         0x29e7c __ulock_wait + 8
1  libdispatch.dylib              0x51fc _dlock_wait + 56
2  libdispatch.dylib              0x4edc _dispatch_wait_on_address + 140
3  libdispatch.dylib              0x55a4 _dispatch_group_wait_slow + 60
4  libdispatch.dylib              0x8880 dispatch_block_wait + 308
5  NovalReader                     0xad9770 -[APMDefaultIdentitySupport resettableDeviceID] + 869904
6  NovalReader                     0xaeda14 __32-[APMIdentity updateIdentifiers]_block_invoke + 952500

If you are not using App Tracking Transparency or AdSupport, you may try using FirebaseAnalyticsWithoutAdIdSupport instead of FirebaseAnalytics.

firebase / firebase-ios-sdk