EXC_BAD_ACCESS on LDClient.get(environment:)

[nicobayer]

Describe the bug App crashes on launch. Stack trace lead to function LDClient.get(environment:).

To reproduce We have been unable to reproduce the crash in a debug environment.

Expected behavior Should not crash.

Logs No logs available.

SDK version 8.0.1

Language version, developer tools Xcode 14.1 / Swift 5

OS/platform Crash only happens on iOS 17.

Additional context Occurrence is very low, less than 0.01% of sessions impacted. Crash happens when the app is woken up to perform a background refresh of data (using BGAppRefreshTask). We believe this is a threading issue where there is multiple threads attempting to use LDClient at the same time to check for feature flag status and potentially other functionality. Are all functions safe to use from any thread?

# Crashlytics - Stack trace
# Application: com.climate.growers.iphone.CGPRelease
# Platform: apple
# Version: 7.4.1 (1)
# Issue: 5876c34020e2a6341e994bc5dd12d19b
# Session: e711b8f0136c42899c10cdb2614df664_DNE_0_v2
# Date: Fri Dec 01 2023 08:46:34 GMT-0300 (Uruguay Standard Time)

com.apple.main-thread
0  libsystem_kernel.dylib         0x2c28 __open + 8
1  libsystem_kernel.dylib         0x2c14 open + 40
2  CoreFoundation                 0x5e934 _CFReadBytesFromFile + 208
3  CoreFoundation                 0x5e250 CFURLCreateDataAndPropertiesFromResource + 324
4  CoreFoundation                 0x5d8b8 _CFBundleCopyInfoDictionaryInDirectoryWithVersion + 1588
5  CoreFoundation                 0x5cf00 _CFBundleRefreshInfoDictionaryAlreadyLocked + 144
6  CoreFoundation                 0x5ce4c CFBundleGetInfoDictionary + 60
7  CoreFoundation                 0x3bdd4 _CFBundleCreate + 488
8  Foundation                     0x331d8 -[NSBundle _cfBundle] + 76
9  Foundation                     0xb800c -[NSBundle resourcePath] + 20
10 CoreUI                         0x60354 __85+[CUIDesignLibraryCatalog catalogForDesignSystem:colorScheme:contrast:styling:error:]_block_invoke + 72
11 libdispatch.dylib              0x4300 _dispatch_client_callout + 20
12 libdispatch.dylib              0x5b3c _dispatch_once_callout + 32
13 CoreUI                         0x1821c +[CUIDesignLibraryCatalog catalogForDesignSystem:colorScheme:contrast:styling:error:] + 640
14 CoreUI                         0x17e38 +[CUIDesignLibraryCompositeCatalog _catalogsForDesignSystem:colorScheme:contrast:styling:error:] + 104
15 CoreUI                         0x17d4c +[CUIDesignLibraryCompositeCatalog compositeCatalogForDesignSystem:colorScheme:contrast:styling:error:] + 172
16 CoreUI                         0x17550 +[CUIDesignLibrary colorWithName:designSystem:palette:colorScheme:contrast:styling:displayGamut:error:] + 120
17 CoreUI                         0x174cc +[CUIDesignLibrary colorWithTraits:error:] + 48
18 UIKitCore                      0x27990 -[UIDynamicCatalogSystemColor _resolvedColorWithTraitCollection:] + 312
19 UIKitCore                      0x44d34 -[UIDynamicColor CGColor] + 60
20 UIKitCore                      0xf4730 -[UIActivityIndicatorView _artCacheKeyWithStyle:color:] + 44
21 UIKitCore                      0xf1428 -[UIActivityIndicatorView layoutSubviews] + 408
22 UIKitCore                      0x32694 -[UIView(CALayerDelegate) layoutSublayersOfLayer:] + 1528
23 QuartzCore                     0x671c0 CA::Layer::layout_if_needed(CA::Transaction*) + 500
24 QuartzCore                     0x66d48 CA::Layer::layout_and_display_if_needed(CA::Transaction*) + 144
25 QuartzCore                     0x6d278 CA::Context::commit_transaction(CA::Transaction*, double, double*) + 464
26 QuartzCore                     0x66574 CA::Transaction::commit() + 648
27 QuartzCore                     0x6621c CA::Transaction::flush_as_runloop_observer(bool) + 88
28 UIKitCore                      0xaa974 _UIApplicationFlushCATransaction + 52
29 UIKitCore                      0xaa48c _UIUpdateSequenceRun + 84
30 UIKitCore                      0xa9b7c schedulerStepScheduledMainSection + 144
31 UIKitCore                      0xa9c38 runloopSourceCallback + 92
32 CoreFoundation                 0x3731c __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 28
33 CoreFoundation                 0x36598 __CFRunLoopDoSource0 + 176
34 CoreFoundation                 0x34d4c __CFRunLoopDoSources0 + 244
35 CoreFoundation                 0x33a88 __CFRunLoopRun + 828
36 CoreFoundation                 0x33668 CFRunLoopRunSpecific + 608
37 GraphicsServices               0x35ec GSEventRunModal + 164
38 UIKitCore                      0x22c2b4 -[UIApplication _run] + 888
39 UIKitCore                      0x22b8f0 UIApplicationMain + 340
40 ClimateGrowers                 0x5398 main + 42 (main.m:42)
41 ???                            0x1b1422dcc (Missing)

com.apple.root.background-qos
0  ???                            0x1b1471200 (Missing)
1  CoreFoundation                 0x10404 _CFExecutableLinkedOnOrAfter + 40
2  CoreFoundation                 0x8bbd4 CFNumberFormatterSetProperty + 1860
3  Foundation                     0xea350 -[NSNumberFormatter setMaximumFractionDigits:] + 196
4  UnitsOfMeasure                 0x58a0 specialized FVDisplayUnit.init(name:unit:precision:displayUnitManager:) + 37 (FVDisplayUnit.swift:37)
5  UnitsOfMeasure                 0x11cb4 specialized FVDisplayUnitManager.init(countryCode:bundle:unitConverter:locale:validate:) + 49 (FVDisplayUnitManager.swift:49)
6  UnitsOfMeasure                 0x6f40 UnitsOfMeasure.resetForCountryCode(countryCode:) + 4804 (FVDisplayUnitManager.swift:4804)
7  ClimateGrowers                 0x174a50 CGPAppDelegate.didLogInUser() + 705 (CGPAppDelegate.swift:705)
8  ClimateGrowers                 0x17bd24 partial apply for closure #1 in closure #1 in CGPAppDelegate.application(_:willFinishLaunchingWithOptions:) + 62 (CGPAppDelegate.swift:62)
9  ClimateGrowers                 0x116c10 thunk for @escaping @callee_guaranteed () -> () + 4364299280 (<compiler-generated>:4364299280)
10 libdispatch.dylib              0x26a8 _dispatch_call_block_and_release + 32
11 libdispatch.dylib              0x4300 _dispatch_client_callout + 20
12 libdispatch.dylib              0x15dbc _dispatch_root_queue_drain + 864
13 libdispatch.dylib              0x163ec _dispatch_worker_thread2 + 156
14 libsystem_pthread.dylib        0x1928 _pthread_wqthread + 228
15 libsystem_pthread.dylib        0x1a04 start_wqthread + 8

Thread
0  libsystem_pthread.dylib        0x19fc start_wqthread + 438

Crashed: com.apple.BGTaskScheduler (com.climate.growers.refresh)
0  LaunchDarkly                   0x50458 static LDClient.get(environment:) + 709 (LDClient.swift:709)
1  ClimateFeatureFlags            0x90b4 protocol witness for FeatureFlagProvider.isFeatureEnabled(_:defaultValue:) in conformance LaunchDarklyFeatureFlagClient + 63 (LaunchDarklyFeatureFlagClient.swift:63)
2  ClimateFeatureFlags            0x7b70 protocol witness for FeatureFlagProvider.isFeatureEnabled(_:defaultValue:) in conformance FeatureFlagService + 76 (FeatureFlagService.swift:76)
3  OfflineSync                    0x5e7c SyncRegistrar.syncSettings.getter + 40 (BackgroundSyncSettingsManager.swift:40)
4  OfflineSync                    0x63d4 closure #3 in SyncRegistrar.().init() + 66 (SyncRegistrar.swift:66)
5  OfflineSync                    0x47274 SyncabilityManager.areBackgroundTasksEnabled() + 70 (SyncabilityManager.swift:70)
6  OfflineSync                    0x4738c protocol witness for SyncabilityManagerType.areBackgroundTasksEnabled() in conformance SyncabilityManager + 43932 (<compiler-generated>:43932)
7  OfflineSync                    0x24248 OfflineSyncManager.performBackgroundSync(_:) + 66 (OfflineSyncManager.swift:66)
8  ClimateGrowers                 0x177380 CGPAppDelegate.handleAppRefresh(task:) + 189 (OfflineSyncSetup.swift:189)
9  ClimateGrowers                 0x17b798 partial apply for closure #4 in CGPAppDelegate.application(_:didFinishLaunchingWithOptions:) + 174 (CGPAppDelegate.swift:174)
10 ClimateGrowers                 0x176c14 thunk for @escaping @callee_guaranteed (@guaranteed BGTask) -> () + 4364692500 (<compiler-generated>:4364692500)
11 BackgroundTasks                0x3bbc __41-[BGTaskScheduler _runTask:registration:]_block_invoke.91 + 196
12 libdispatch.dylib              0x26a8 _dispatch_call_block_and_release + 32
13 libdispatch.dylib              0x4300 _dispatch_client_callout + 20
14 libdispatch.dylib              0xb894 _dispatch_lane_serial_drain + 748
15 libdispatch.dylib              0xc3f8 _dispatch_lane_invoke + 432
16 libdispatch.dylib              0x17004 _dispatch_root_queue_drain_deferred_wlh + 288
17 libdispatch.dylib              0x16878 _dispatch_workloop_worker_thread + 404
18 libsystem_pthread.dylib        0x1964 _pthread_wqthread + 288
19 libsystem_pthread.dylib        0x1a04 start_wqthread + 8

com.apple.uikit.eventfetch-thread
0  libsystem_kernel.dylib         0x11d8 mach_msg2_trap + 8
1  libsystem_kernel.dylib         0xf70 mach_msg2_internal + 80
2  libsystem_kernel.dylib         0xe88 mach_msg_overwrite + 436
3  libsystem_kernel.dylib         0xcc8 mach_msg + 24
4  CoreFoundation                 0x35d0c __CFRunLoopServiceMachPort + 160
5  CoreFoundation                 0x33c04 __CFRunLoopRun + 1208
6  CoreFoundation                 0x33668 CFRunLoopRunSpecific + 608
7  Foundation                     0x2c54c -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 212
8  Foundation                     0x5a27c -[NSRunLoop(NSRunLoop) runUntilDate:] + 64
9  UIKitCore                      0x18dfc8 -[UIEventFetcher threadMain] + 420
10 Foundation                     0xb1184 __NSThread__start__ + 732
11 libsystem_pthread.dylib        0x24d4 _pthread_start + 136
12 libsystem_pthread.dylib        0x1a10 thread_start + 8

Thread
0  libsystem_pthread.dylib        0x19fc start_wqthread + 438

com.launchdarkly.FlagSynchronizer.isOnlineQueue
0  LDSwiftEventSource             0xcdf8 type metadata accessor for EventSource.Config + 18
1  LaunchDarkly                   0x44b28 specialized ClientServiceFactory.makeStreamingProvider(url:httpHeaders:connectMethod:connectBody:handler:delegate:errorHandler:) + 68 (ClientServiceFactory.swift:68)
2  LaunchDarkly                   0x44770 specialized DarklyService.createEventSource(useReport:handler:errorHandler:) + 170 (DarklyService.swift:170)
3  LaunchDarkly                   0x3f2b0 FlagSynchronizer.startEventSource() + 125 (FlagSynchronizer.swift:125)
4  LaunchDarkly                   0x3e79c closure #1 in FlagSynchronizer.isOnline.setter + 73 (FlagSynchronizer.swift:73)
5  LaunchDarkly                   0x48bb0 thunk for @callee_guaranteed () -> () + 20 (<compiler-generated>:20)
6  LaunchDarkly                   0x48bd0 thunk for @escaping @callee_guaranteed () -> () + 20 (<compiler-generated>:20)
7  libdispatch.dylib              0x4300 _dispatch_client_callout + 20
8  libdispatch.dylib              0x136b4 _dispatch_lane_barrier_sync_invoke_and_complete + 56
9  LaunchDarkly                   0x3e4e4 FlagSynchronizer.isOnline.setter + 69 (FlagSynchronizer.swift:69)
10 LaunchDarkly                   0x44114 protocol witness for LDFlagSynchronizing.isOnline.setter in conformance FlagSynchronizer + 20 (<compiler-generated>:20)
11 LaunchDarkly                   0x49998 closure #1 in LDClient.isOnline.setter + 63 (LDClient.swift:63)
12 LaunchDarkly                   0x54e24 partial apply for thunk for @callee_guaranteed () -> () + 20 (<compiler-generated>:20)
13 LaunchDarkly                   0x48bd0 thunk for @escaping @callee_guaranteed () -> () + 20 (<compiler-generated>:20)
14 libdispatch.dylib              0x4300 _dispatch_client_callout + 20
15 libdispatch.dylib              0x136b4 _dispatch_lane_barrier_sync_invoke_and_complete + 56
16 LaunchDarkly                   0x4a458 LDClient.go(online:reasonOnlineUnavailable:completion:) + 60 (LDClient.swift:60)
17 LaunchDarkly                   0x58860 partial apply for closure #1 in closure #1 in LDClient.internalSetOnline(_:completion:) + 127 (LDClient.swift:127)
18 LaunchDarkly                   0xaa38 partial apply for thunk for @escaping @callee_guaranteed () -> () + 20 (<compiler-generated>:20)
19 LaunchDarkly                   0xa6b4 thunk for @escaping @callee_guaranteed () -> (@out ()) + 20 (<compiler-generated>:20)
20 LaunchDarkly                   0x75c70 thunk for @escaping @callee_guaranteed () -> () + 28 (<compiler-generated>:28)
21 libdispatch.dylib              0x26a8 _dispatch_call_block_and_release + 32
22 libdispatch.dylib              0x4300 _dispatch_client_callout + 20
23 libdispatch.dylib              0x15dbc _dispatch_root_queue_drain + 864
24 libdispatch.dylib              0x163ec _dispatch_worker_thread2 + 156
25 libsystem_pthread.dylib        0x1928 _pthread_wqthread + 228
26 libsystem_pthread.dylib        0x1a04 start_wqthread + 8

Thread
0  libsystem_pthread.dylib        0x19fc start_wqthread + 438

Thread
0  libsystem_pthread.dylib        0x19fc start_wqthread + 438

com.google.firebase.crashlytics.MachExceptionServer
0  FirebaseCrashlytics            0x1e790 FIRCLSProcessRecordAllThreads + 393 (FIRCLSProcess.c:393)
1  FirebaseCrashlytics            0x1eb70 FIRCLSProcessRecordAllThreads + 424 (FIRCLSProcess.c:424)
2  FirebaseCrashlytics            0x16c38 FIRCLSHandler + 34 (FIRCLSHandler.m:34)
3  FirebaseCrashlytics            0x195cc FIRCLSMachExceptionServer + 524 (FIRCLSMachException.c:524)
4  libsystem_pthread.dylib        0x24d4 _pthread_start + 136
5  libsystem_pthread.dylib        0x1a10 thread_start + 8

Thread
0  libsystem_kernel.dylib         0x19d8 __semwait_signal + 8
1  libsystem_c.dylib              0x9f20 nanosleep + 220
2  Foundation                     0x74c6cc +[NSThread sleepForTimeInterval:] + 160
3  ClimateGrowers                 0x272930 WatcherThread.main() + 78 (WatchCat.swift:78)
4  ClimateGrowers                 0x272a18 @objc WatcherThread.main() + 4365724184 (<compiler-generated>:4365724184)
5  Foundation                     0xb1184 __NSThread__start__ + 732
6  libsystem_pthread.dylib        0x24d4 _pthread_start + 136
7  libsystem_pthread.dylib        0x1a10 thread_start + 8

Thread
0  libsystem_kernel.dylib         0x19d8 __semwait_signal + 8
1  libsystem_c.dylib              0x9f20 nanosleep + 220
2  Foundation                     0x74c6cc +[NSThread sleepForTimeInterval:] + 160
3  ClimateGrowers                 0x272930 WatcherThread.main() + 78 (WatchCat.swift:78)
4  ClimateGrowers                 0x272a18 @objc WatcherThread.main() + 4365724184 (<compiler-generated>:4365724184)
5  Foundation                     0xb1184 __NSThread__start__ + 732
6  libsystem_pthread.dylib        0x24d4 _pthread_start + 136
7  libsystem_pthread.dylib        0x1a10 thread_start + 8

[tanderson-ld]

Hi @nicobayer, thank you for taking the time to report this issue. It is especially helpful to have the stack trace.

LDClient.get(...) should be thread safe. All methods on a given instance of an LDClient are also thread safe. I suspect this has less to do with threading and more to do with a held memory reference that the OS cleaned up while in the background. I'm investigating possible causes. Do you ever hold a reference to an LDClient returned from LDClient.get(...) any where in your code? If so, could you describe the lifecycle of that reference and the class holding that reference.

How many total occurrences have you seen in your crash monitoring tool?

Thank you!

[nicobayer]

@tanderson-ld we don't hold a reference to LDClient, we always go with LDClient.get() to get an instance.

We had 449 occurrences of this crash in the last 90 days.

I'm still looking into this but my guess is that LDClient.start() happens at the same time as we are checking for a feature flag with LDClient.get().boolVariation(forKey:defaultValue:).

Will come back with additional information if I find anything else.

Thanks for your prompt response :)

[tanderson-ld]

That sounds like it could be a likely culprit. start(...) needs to complete before flag evaluation calls are made when the SDK is not initialized, which may be the case coming from a background state. You can check isInitialized to help debug.

If you have multiple threads, you may need to set up synchronization between start(...) completing and the thread doing the evaluation. There is a flavor of start(...) which supports a timeout. You can think of it as "start waiting up to this amount of time to get fresh flag data from the cloud, if timeout hit, proceed with cache flag data but still try to get fresh data". In any flavor you call in a multithreaded situation, you should wait for the completion callback before making the flag evaluation call.

[nicobayer]

I see, we are definitely not waiting for start() to end before evaluating flags so will look for ways to make this happen.

[tanderson-ld]

Are you calling start(...) when your app launches normally (not from the background)? Perhaps that app launch code path is single threaded and why you don't see the crash in clean launch / foreground usage cases?

Note: Keep in mind that if you structure your start(...) to not wait for fresh flag data (perhaps launch time is super important), on first launch this may result in default flag values. There is really no perfect solution to that situation and is best handled by reactive programming so that when flag values are eventually fetched your user experience updates dynamically.

[nicobayer]

Correct, start() is called in initial app launch event, not in background/foreground events.

We do not wait for LD to finish start() call as we want to reduce risk of LD bottlenecking our startup process. As you said tradeoff would be getting outdated flag states which we are okay with as some of our UI refreshes on appear events.

We ended up adding a concurrent queue to synchronize LD usage (snippet below). We took a peak at the LDClient.swift and noticed LDClient.instances private dictionary, swift collections are not thread-safe so we think that is our root cause for the crash. Probably not the only solution but we feel like this should fix the crash and also allow us to continue using LD safely from any thread.

I'm going to close the ticket but feel free to re-open if you want to continue discussion. We will know if this fixed the crash a few months from now as we have a long release cycle.

Thanks!

let syncQueue = DispatchQueue(label: "com.climate.launchDarklySyncQueue", qos: .userInitiated, attributes: .concurrent)

func getClient() -> LDClient? {
    var client: LDClient?
    syncQueue.sync {
        client = LDClient.get()
    }
    return client
}

func start(user: FeatureFlagsUser?, completion: FeatureFlagStartCompletion?) {
    syncQueue.async(flags: .barrier) {
        LDClient.start(...) { 
                DispatchQueue.main.async { completion?(...) }
        }
    }
}