search

firebase / firebase-ios-sdk

Firebase SDK for Apple App Development
https://firebase.google.com
Apache License 2.0
5.59k stars 1.46k forks source link

Segmentation Fault: 11 after upgrading past 6.32.0 #6734

Closed tspecht closed 3 years ago

tspecht commented 3 years ago

[REQUIRED] Step 1: Describe your environment

Output of cat Podfile.lock | grep Firebase:

- Firebase/Analytics (6.34.0):
    - Firebase/Core
  - Firebase/AppDistribution (6.34.0):
    - Firebase/CoreOnly
    - FirebaseAppDistribution (~> 0.9.3)
  - Firebase/Auth (6.34.0):
    - Firebase/CoreOnly
    - FirebaseAuth (~> 6.9.2)
  - Firebase/Core (6.34.0):
    - Firebase/CoreOnly
    - FirebaseAnalytics (= 6.9.0)
  - Firebase/CoreOnly (6.34.0):
    - FirebaseCore (= 6.10.4)
  - Firebase/Crashlytics (6.34.0):
    - Firebase/CoreOnly
    - FirebaseCrashlytics (~> 4.6.2)
  - Firebase/Performance (6.34.0):
    - Firebase/CoreOnly
    - FirebasePerformance (~> 3.3.1)
  - Firebase/RemoteConfig (6.34.0):
    - Firebase/CoreOnly
    - FirebaseRemoteConfig (~> 4.9.1)
  - FirebaseABTesting (4.2.0):
    - FirebaseCore (~> 6.10)
  - FirebaseAnalytics (6.9.0):
    - FirebaseCore (~> 6.10)
    - FirebaseInstallations (~> 1.7)
  - FirebaseAppDistribution (0.9.3):
    - FirebaseCore (~> 6.10)
    - FirebaseInstallations (~> 1.6)
  - FirebaseAuth (6.9.2):
    - FirebaseCore (~> 6.10)
  - FirebaseCore (6.10.4):
    - FirebaseCoreDiagnostics (~> 1.6)
  - FirebaseCoreDiagnostics (1.7.0):
  - FirebaseCrashlytics (4.6.2):
    - FirebaseCore (~> 6.10)
    - FirebaseInstallations (~> 1.6)
  - FirebaseInstallations (1.7.0):
    - FirebaseCore (~> 6.10)
  - FirebasePerformance (3.3.1):
    - FirebaseCore (~> 6.9)
    - FirebaseInstallations (~> 1.5)
    - FirebaseRemoteConfig (~> 4.7)
  - FirebaseRemoteConfig (4.9.1):
    - FirebaseABTesting (~> 4.2)
    - FirebaseCore (~> 6.10)
    - FirebaseInstallations (~> 1.6)
  - Firebase/Analytics (= 6.34.0)
  - Firebase/AppDistribution (= 6.34.0)
  - Firebase/Auth (= 6.34.0)
  - Firebase/Crashlytics (= 6.34.0)
  - Firebase/Performance (= 6.34.0)
  - Firebase/RemoteConfig (= 6.34.0)
    - Firebase
    - FirebaseABTesting
    - FirebaseAnalytics
    - FirebaseAppDistribution
    - FirebaseAuth
    - FirebaseCore
    - FirebaseCoreDiagnostics
    - FirebaseCrashlytics
    - FirebaseInstallations
    - FirebasePerformance
    - FirebaseRemoteConfig
  Firebase: c23a36d9e4cdf7877dfcba8dd0c58add66358999
  FirebaseABTesting: 8a9d8df3acc2b43f4a22014ddf9f601bca6af699
  FirebaseAnalytics: 3bb096873ee0d7fa4b6c70f5e9166b6da413cc7f
  FirebaseAppDistribution: 3b615c5fba5953351c5c96ca03217d15b0ee85d2
  FirebaseAuth: c92d49ada7948d1a23466e3db17bc4c2039dddc3
  FirebaseCore: d3a978a3cfa3240bf7e4ba7d137fdf5b22b628ec
  FirebaseCoreDiagnostics: 770ac5958e1372ce67959ae4b4f31d8e127c3ac1
  FirebaseCrashlytics: 1a747c9cc084a24dc6d9511c991db1cd078154eb
  FirebaseInstallations: 466c7b4d1f58fe16707693091da253726a731ed2
  FirebasePerformance: e325a8ee84a6a3d89c0be049390ed6c1775cce22
  FirebaseRemoteConfig: 35a729305f254fb15a2e541d4b36f3a379da7fdc

[REQUIRED] Step 2: Describe the problem

After upgrading to Firebase 6.32.2 or later from 6.32.0, we get intermittent crashes in what seems to be the iOS internal networking stack. We are seeing Segmentation Fault: 11 crashes in networking-related code within Alamofire, URLSession as well as Kingfisher, all without updating any of the SDKs. When downgrading to 6.32.0 again everything works fine, so it's related to Firebase.

Steps to reproduce:

The issue happens rather randomly during what used to be normal usage of the app, but seems to be consistently related to networking code looking at the crash logs.

I attached two crash logs:

crash_report_1.txt crash_report_2.txt

morganchen12 commented 3 years ago

I suspect Performance is doing something bad. Can you try excluding Performance?

maksymmalyhin commented 3 years ago

@tspecht Sorry you have the issue.

From the first glance the crash indicates a concurrent memory access somewhere around NSBlockOperation or memory management issue. The issue may be exposed by changes in Firebase Performance SDK but it would not necessarily mean that Firebase Performance is the root reason. Options I would suggest to try:

  1. Run your app/tests with Thread, Address sanitizers and Zombie Objects: Screen Shot 2020-10-15 at 12 12 53 PM
  2. Try to instrumentation for Firebase Performance

Please share the results so we can think of the next steps.

Note: It also may be related to #6713.

tspecht commented 3 years ago

@maksymmalyhin I did try running with the different checkers, unfortunately nothing interesting came out of this. Not sure what you mean by point #2?

maksymmalyhin commented 3 years ago

@tspecht I mean setting isInstrumentationEnabled. It allows you to keep custom performance traces(if you use them) but disable metrics collected automatically that imply swizzling, etc.

tspecht commented 3 years ago

@maksymmalyhin currently running our CI steps a bunch of times which takes a while, first pass with isInstrumentationEnabled = false has been looking good though, so might have fixed the crashes. Will post another update in ~2h

maksymmalyhin commented 3 years ago

@tspecht Thank you!

tspecht commented 3 years ago

@maksymmalyhin ran it two additional times and still seeing crashes:

crash_log_3.txt

maksymmalyhin commented 3 years ago

@tspecht is the crash with isInstrumentationEnabled = false ?

tspecht commented 3 years ago

@maksymmalyhin unfortunately yes, calling the following as part of my AppDelegate.didFinishLaunching:

Performance.sharedInstance().isInstrumentationEnabled = false
maksymmalyhin commented 3 years ago

@tspecht It most likely indicates that the crash is not related to Performance SDK. Just to be more sure on it, could you please test with Firebase Performance excluded from your app?

tspecht commented 3 years ago

@maksymmalyhin we just finished running 5 builds in a row without the Firebase/Performance pod.

Previously we used to have a failure rate of roughly 2 out of 3 builds failing due to the crash mentioned above. After removing Firebase/Performance we ran 5 builds in a row successfully without a single crash.

While this isn't proof, I think statistically speaking this indicates that Firebase/Performance really is to blame here.

jimmyhoran commented 3 years ago

We ran into the same issue, so we had started exhaustively testing our builds to try an establish a trend. We were successful in semi-reliably reproducing the crash within 5 attempts of uninstalling, re-installing and launching the same application build.

We had recently updated Firebase/Crashlytics, Firebase/Analytics and Firebase/Performance from 6.31.0 to 6.34.0. After reverting back to 6.31.0, I can confidently say the issue is resolved for us.

If there's any additional information someone from Firebase needs to look into this, please let me know. I'd be happy to provide any additional information to aid this being resolved in a future version 🙏

Kaspik commented 3 years ago

We are seeing thousands of crashes on our latest app version due to Heap Corruption and we can't figure why as the logs are pretty unusable. We updated our app from 6.33 to 6.34 (6.33 was okay), here is a diff of our Podfile.lock if it helps:

  - Firebase/Analytics (6.33.0):          - Firebase/Analytics (6.34.0):
    - Firebase/Core         - Firebase/Core
  - Firebase/Core (6.33.0):       - Firebase/Core (6.34.0):
    - Firebase/CoreOnly         - Firebase/CoreOnly
    - FirebaseAnalytics (= 6.8.3)           - FirebaseAnalytics (= 6.9.0)
  - Firebase/CoreOnly (6.33.0):       - Firebase/CoreOnly (6.34.0):
    - FirebaseCore (= 6.10.3)           - FirebaseCore (= 6.10.4)
  - Firebase/Crashlytics (6.33.0):        - Firebase/Crashlytics (6.34.0):
    - Firebase/CoreOnly         - Firebase/CoreOnly
    - FirebaseCrashlytics (~> 4.6.1)            - FirebaseCrashlytics (~> 4.6.2)
  - Firebase/Messaging (6.33.0):          - Firebase/Messaging (6.34.0):
    - Firebase/CoreOnly         - Firebase/CoreOnly
    - FirebaseMessaging (~> 4.7.0)          - FirebaseMessaging (~> 4.7.1)
  - Firebase/Performance (6.33.0):        - Firebase/Performance (6.34.0):
    - Firebase/CoreOnly         - Firebase/CoreOnly
    - FirebasePerformance (~> 3.3.0)            - FirebasePerformance (~> 3.3.1)
  - FirebaseABTesting (4.2.0):        - FirebaseABTesting (4.2.0):
    - FirebaseCore (~> 6.10)            - FirebaseCore (~> 6.10)
  - FirebaseAnalytics (6.8.3):        - FirebaseAnalytics (6.9.0):
    - FirebaseCore (~> 6.10)            - FirebaseCore (~> 6.10)
    - FirebaseInstallations (~> 1.6)            - FirebaseInstallations (~> 1.7)
    - GoogleAppMeasurement (= 6.8.3)            - GoogleAppMeasurement (= 6.9.0)
    - GoogleUtilities/AppDelegateSwizzler (~> 6.7)          - GoogleUtilities/AppDelegateSwizzler (~> 6.7)
    - GoogleUtilities/MethodSwizzler (~> 6.7)           - GoogleUtilities/MethodSwizzler (~> 6.7)
    - GoogleUtilities/Network (~> 6.7)          - GoogleUtilities/Network (~> 6.7)
    - "GoogleUtilities/NSData+zlib (~> 6.7)"            - "GoogleUtilities/NSData+zlib (~> 6.7)"
    - nanopb (~> 1.30906.0)         - nanopb (~> 1.30906.0)
  - FirebaseCore (6.10.3):        - FirebaseCore (6.10.4):
    - FirebaseCoreDiagnostics (~> 1.6)          - FirebaseCoreDiagnostics (~> 1.6)
    - GoogleUtilities/Environment (~> 6.7)          - GoogleUtilities/Environment (~> 6.7)
    - GoogleUtilities/Logger (~> 6.7)           - GoogleUtilities/Logger (~> 6.7)
 @@ -56,7 +56,7 @@ PODS:
    - GoogleUtilities/Environment (~> 6.7)          - GoogleUtilities/Environment (~> 6.7)
    - GoogleUtilities/Logger (~> 6.7)           - GoogleUtilities/Logger (~> 6.7)
    - nanopb (~> 1.30906.0)         - nanopb (~> 1.30906.0)
  - FirebaseCrashlytics (4.6.1):          - FirebaseCrashlytics (4.6.2):
    - FirebaseCore (~> 6.10)            - FirebaseCore (~> 6.10)
    - FirebaseInstallations (~> 1.6)            - FirebaseInstallations (~> 1.6)
    - GoogleDataTransport (~> 7.2)          - GoogleDataTransport (~> 7.2)
 @@ -67,20 +67,20 @@ PODS:
    - GoogleUtilities/Environment (~> 6.7)          - GoogleUtilities/Environment (~> 6.7)
    - GoogleUtilities/UserDefaults (~> 6.7)         - GoogleUtilities/UserDefaults (~> 6.7)
    - PromisesObjC (~> 1.2)         - PromisesObjC (~> 1.2)
  - FirebaseInstanceID (4.7.0):       - FirebaseInstanceID (4.8.0):
    - FirebaseCore (~> 6.10)            - FirebaseCore (~> 6.10)
    - FirebaseInstallations (~> 1.6)            - FirebaseInstallations (~> 1.6)
    - GoogleUtilities/Environment (~> 6.7)          - GoogleUtilities/Environment (~> 6.7)
    - GoogleUtilities/UserDefaults (~> 6.7)         - GoogleUtilities/UserDefaults (~> 6.7)
  - FirebaseMessaging (4.7.0):        - FirebaseMessaging (4.7.1):
    - FirebaseCore (~> 6.10)            - FirebaseCore (~> 6.10)
    - FirebaseInstanceID (~> 4.7)           - FirebaseInstanceID (~> 4.7)
    - GoogleUtilities/AppDelegateSwizzler (~> 6.7)          - GoogleUtilities/AppDelegateSwizzler (~> 6.7)
    - GoogleUtilities/Environment (~> 6.7)          - GoogleUtilities/Environment (~> 6.7)
    - GoogleUtilities/Reachability (~> 6.7)         - GoogleUtilities/Reachability (~> 6.7)
    - GoogleUtilities/UserDefaults (~> 6.7)         - GoogleUtilities/UserDefaults (~> 6.7)
    - Protobuf (>= 3.9.2, ~> 3.9)           - Protobuf (>= 3.9.2, ~> 3.9)
  - FirebasePerformance (3.3.0):          - FirebasePerformance (3.3.1):
    - FirebaseCore (~> 6.9)         - FirebaseCore (~> 6.9)
    - FirebaseInstallations (~> 1.5)            - FirebaseInstallations (~> 1.5)
    - FirebaseRemoteConfig (~> 4.7)         - FirebaseRemoteConfig (~> 4.7)
 @@ -92,20 +92,20 @@ PODS:
    - GoogleUtilities/MethodSwizzler (~> 6.2)           - GoogleUtilities/MethodSwizzler (~> 6.2)
    - GTMSessionFetcher/Core (~> 1.1)           - GTMSessionFetcher/Core (~> 1.1)
    - Protobuf (~> 3.12)            - Protobuf (~> 3.12)
  - FirebaseRemoteConfig (4.9.0):         - FirebaseRemoteConfig (4.9.1):
    - FirebaseABTesting (~> 4.2)            - FirebaseABTesting (~> 4.2)
    - FirebaseCore (~> 6.10)            - FirebaseCore (~> 6.10)
    - FirebaseInstallations (~> 1.6)            - FirebaseInstallations (~> 1.6)
    - GoogleUtilities/Environment (~> 6.7)          - GoogleUtilities/Environment (~> 6.7)
    - "GoogleUtilities/NSData+zlib (~> 6.7)"            - "GoogleUtilities/NSData+zlib (~> 6.7)"
  - GoogleAppMeasurement (6.8.3):         - GoogleAppMeasurement (6.9.0):
    - GoogleUtilities/AppDelegateSwizzler (~> 6.7)          - GoogleUtilities/AppDelegateSwizzler (~> 6.7)
    - GoogleUtilities/MethodSwizzler (~> 6.7)           - GoogleUtilities/MethodSwizzler (~> 6.7)
    - GoogleUtilities/Network (~> 6.7)          - GoogleUtilities/Network (~> 6.7)
    - "GoogleUtilities/NSData+zlib (~> 6.7)"            - "GoogleUtilities/NSData+zlib (~> 6.7)"
    - nanopb (~> 1.30906.0)         - nanopb (~> 1.30906.0)
  - GoogleDataTransport (7.4.0):          - GoogleDataTransport (7.5.1):
    - nanopb (~> 1.30906.0)         - nanopb (~> 1.30906.0)
  - GoogleToolboxForMac/Defines (2.2.2)       - GoogleToolboxForMac/Defines (2.2.2)
  - GoogleToolboxForMac/Logger (2.2.2):       - GoogleToolboxForMac/Logger (2.2.2):

We do have Performance monitoring included, I'm going to remove it now if it helps and let you know, I just thought I'll leave this here as well if it helps debugging this issue.

EDIT: Part of Performance monitoring is also ISASwizzler, MethodSwizzler and other libs (via screenshot) - it might be some of them as well. ISASwizzler was refactored couple of days ago to support ARC (https://github.com/firebase/firebase-ios-sdk/pull/5862), but I don't think this was part of 6.34 yet

Screenshot 2020-10-16 at 11 26 32
maksymmalyhin commented 3 years ago

Thank you for the provided details. We will continue investigation and post updates to the thread.

davidgrigoryan commented 3 years ago

@maksymmalyhin Hello. We are receiving same crashes from Alamofire after updating Firebase SDK to the last version. Would you suggest to downgrade to previous version and update the app before you fix the issue?

visumickey commented 3 years ago

Thanks for the report. In the most recent version of Firebase performance, we started using KVO to measure the completion of the network request. Not sure if this is conflicting with Alamofire. Can someone provide the following details?

  1. Version of Alamofire used by your app.
  2. Type of network request where this crash is occurring (Eg. DataTaskWithCompletionHandler, dataTaskWithURLRequest)
  3. Any simple/sample app where we can reproduce the issue? (Even steps would do!)

We have already fixed a potential vulnerable code and this is expected to release soon. My guess is that this issue should be resolved with the fix. Will keep the thread posted.

Kaspik commented 3 years ago

I would love to provide you with all the crashes we are seeing, but Firebase console doesn't allow us to share crashes as Fabric did, nor it has a way to export them.

To answer: 1) Latest release from July - 5.2.2 (We also use AFNetworking that we are migrating from, but we have this mixed project for months already) 2) Not sure I can say specifically 3) Don't have one right now, sorry!

If that helps, we also use PINRemoteImage for remote images loading, and we are seeing the crash there as well.

Here is a txt file with 4 top crashes (we are talking about thousands of crashes on 1% rollout) - always only the first important thread, not the rest of threads. I hope that helps!

crashes.txt

tspecht commented 3 years ago

@visumickey we are also using Alamofire 5.2.2. We have also observed the same exact crash pattern but when using networking from within Kingfisher as well as using plain-old URLSession itself - not sure how likely it is that this is just a bad synergy with Alamofire specifically.

Kaspik commented 3 years ago

We released new version of our app with Firebase 6.34.0 still but without Firebase Performance - no crashes so far, everything seems okay. That means the problem is in Firebase Performance and wasn't present in 6.33.0. :)

tspecht commented 3 years ago

@visumickey any update on this? Inclined to just remove the SDK for now, this is too big of a risk for us.

ejensen commented 3 years ago

After removing the Firebase Performance SDK, all of these crashes stopped. It's safe to say the source of the crashes is in the Firebase code.

visumickey commented 3 years ago

@tspecht We have added a potential fix to this issue in the upcoming release expected to go out in the next few days. In the meanwhile, we will also to continue to see if there are any other defensive coding we can bring in to avoid such issues.

The root cause of the issue is that we introduced a KVO based identification of the network request completeness and that seem to have some conflicts.

willbattel commented 3 years ago

@paulb777 I see that this fix is going in the major v7 update- any chance we can also get it in a v6 patch for those not immediately upgrading to v7? Our project was going to wait a little while before making the jump for various reasons.

Kaspik commented 3 years ago

The same here - we are still migrating from directChannel in Messaging and pushes, and we won't be able to upgrade to Firebase 7 immediately due to this.

cythb commented 3 years ago

I confirm that I find this crash since I have updated the firebase and don't update the Alamofire. The crashes are gone when I downgrade the firebase from 5.2.2.

paulb777 commented 3 years ago

@willbattel @Kaspik @visumickey We're investigating the possibility of releasing a CocoaPods FirebasePerformance 3.3.2 patch to address this issue for Firebase 6.x. We'll follow up here as we flush out the plan.

raymondlam commented 3 years ago

We have released FirebasePerformance 3.3.2 which reverted the KVO changes. Can you update FirebasePerformance to 3.3.2 and see if that fixes the issue?

We are planning to bring back KVO (to handle network request completion) along with the potential fix for Firebase 7.x.x.

paulb777 commented 3 years ago

Closing now that the fix is out in FirebasePerformance 3.3.2.

willbattel commented 3 years ago

Thanks! Appreciate it.

tspecht commented 3 years ago

@paulb777 we just upgraded Firebase to 7.0.0 and are still seeing these issues on our CI, if any more prominent than it used to before..

Process:               MyApp [7525]
Path:                  /Users/USER/Library/Developer/CoreSimulator/Devices/3794BC99-9345-4C64-9C76-358D2F23DE09/data/Containers/Bundle/Application/B7212CCD-748B-4586-B143-43AC41DF3539/MyApp.app/MyApp
Identifier:            MyApp
Version:               5.16.0 (42595)
Code Type:             X86-64 (Native)
Parent Process:        launchd_sim [2313]
Responsible:           SimulatorTrampoline [2222]
User ID:               501

Date/Time:             2020-10-28 16:46:29.869 +0000
OS Version:            Mac OS X 10.15.5 (19F101)
Report Version:        12
Anonymous UUID:        4763FA26-E06C-409D-9FAC-0174438B0AC7

Time Awake Since Boot: 1000 seconds

System Integrity Protection: disabled

Crashed Thread:        10  Dispatch queue: com.apple.NSURLSession-delegate

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       EXC_I386_GPFLT
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [7525]

Application Specific Information:
CoreSimulator 732.17 - Device: iPhone 11 (********-****-****-****-************) - Runtime: iOS 14.0 (18A372) - DeviceType: iPhone 11

Thread 0:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib          0x000000011cd67dfa mach_msg_trap + 10
1   libsystem_kernel.dylib          0x000000011cd68170 mach_msg + 60
2   com.apple.CoreFoundation        0x0000000119ac8e77 __CFRunLoopServiceMachPort + 316
3   com.apple.CoreFoundation        0x0000000119ac358d __CFRunLoopRun + 1284
4   com.apple.CoreFoundation        0x0000000119ac2b9e CFRunLoopRunSpecific + 567
5   com.apple.GraphicsServices      0x00000001226b7db3 GSEventRunModal + 139
6   com.apple.UIKitCore             0x0000000123bc3af3 -[UIApplication _run] + 912
7   com.apple.UIKitCore             0x0000000123bc8a04 UIApplicationMain + 101
8   com.mycompany.myapp             0x000000010c5cc52b main + 75 (AppDelegate.swift:23)
9   libdyld.dylib                   0x000000011c98f415 start + 1

Thread 1:
0   libsystem_pthread.dylib         0x000000011cdc7b68 start_wqthread + 0

Thread 2:: Dispatch queue: com.apple.NSURLSession-work
0   com.apple.CoreFoundation        0x0000000119afdf98 CFStringFindWithOptionsAndLocale + 10528
1   com.apple.CoreFoundation        0x0000000119b01315 CFStringFind + 78
2   com.apple.CFNetwork             0x0000000112d33e39 0x112b55000 + 1961529
3   com.apple.CFNetwork             0x0000000112be7e2d 0x112b55000 + 601645
4   com.apple.CFNetwork             0x0000000112cd2748 0x112b55000 + 1562440
5   com.apple.CFNetwork             0x0000000112bebc7d 0x112b55000 + 617597
6   libdispatch.dylib               0x000000011c9136bb _dispatch_block_async_invoke2 + 83
7   libdispatch.dylib               0x000000011c906534 _dispatch_client_callout + 8
8   libdispatch.dylib               0x000000011c90c425 _dispatch_lane_serial_drain + 715
9   libdispatch.dylib               0x000000011c90cfd7 _dispatch_lane_invoke + 458
10  libdispatch.dylib               0x000000011c917591 _dispatch_workloop_worker_thread + 782
11  libsystem_pthread.dylib         0x000000011cdc8a3d _pthread_wqthread + 290
12  libsystem_pthread.dylib         0x000000011cdc7b77 start_wqthread + 15

Thread 3:
0   libsystem_pthread.dylib         0x000000011cdc7b68 start_wqthread + 0

Thread 4:: Dispatch queue: com.apple.CFNetwork.Connection
0   libsystem_c.dylib               0x000000011ca5810e gettimeofday + 6
1   com.apple.CoreFoundation        0x0000000119b4856f CFAbsoluteTimeGetCurrent + 37
2   com.apple.CFNetwork             0x0000000112c84c08 0x112b55000 + 1244168
3   com.apple.CFNetwork             0x0000000112d6f14d 0x112b55000 + 2203981
4   com.apple.CFNetwork             0x0000000112cfdbf2 0x112b55000 + 1739762
5   com.apple.CFNetwork             0x0000000112d64e62 0x112b55000 + 2162274
6   com.apple.CFNetwork             0x0000000112d67418 0x112b55000 + 2171928
7   com.apple.CFNetwork             0x0000000112d67011 0x112b55000 + 2170897
8   com.apple.CFNetwork             0x0000000112d65ea9 0x112b55000 + 2166441
9   com.apple.CFNetwork             0x0000000112d6667b 0x112b55000 + 2168443
10  com.apple.CFNetwork             0x0000000112d65808 0x112b55000 + 2164744
11  libdispatch.dylib               0x000000011c90535b _dispatch_call_block_and_release + 12
12  libdispatch.dylib               0x000000011c906534 _dispatch_client_callout + 8
13  libdispatch.dylib               0x000000011c90c425 _dispatch_lane_serial_drain + 715
14  libdispatch.dylib               0x000000011c90cfd7 _dispatch_lane_invoke + 458
15  libdispatch.dylib               0x000000011c90e32c _dispatch_workloop_invoke + 1947
16  libdispatch.dylib               0x000000011c917591 _dispatch_workloop_worker_thread + 782
17  libsystem_pthread.dylib         0x000000011cdc8a3d _pthread_wqthread + 290
18  libsystem_pthread.dylib         0x000000011cdc7b77 start_wqthread + 15

Thread 5:
0   libsystem_pthread.dylib         0x000000011cdc7b68 start_wqthread + 0

Thread 6:: Dispatch queue: com.google.GDTCORFlatFileStorage
0   libsystem_kernel.dylib          0x000000011cd686a2 __open + 10
1   com.apple.CoreServicesInternal  0x000000014b188be8 DirEnumRead + 187
2   com.apple.CoreServicesInternal  0x000000014b18519c _GetDirectoryURLs(_CFURLEnumerator*) + 290
3   com.apple.CoreServicesInternal  0x000000014b18441e _URLEnumeratorGetNextURL + 171
4   com.apple.Foundation            0x0000000114b69d73 -[NSURLDirectoryEnumerator nextObject] + 86
5   com.apple.CoreFoundation        0x0000000119b5a5ec -[NSEnumerator countByEnumeratingWithState:objects:count:] + 51
6   org.cocoapods.GoogleDataTransport   0x0000000115113269 -[GDTCORDirectorySizeTracker calculateDirectoryContentSize] + 650
7   org.cocoapods.GoogleDataTransport   0x0000000115112d9b -[GDTCORDirectorySizeTracker directoryContentSize] + 82 (GDTCORDirectorySizeTracker.m:41)
8   org.cocoapods.GoogleDataTransport   0x0000000115115404 __47-[GDTCORFlatFileStorage storeEvent:onComplete:]_block_invoke.60 + 409 (GDTCORFlatFileStorage.m:152)
9   libdispatch.dylib               0x000000011c90535b _dispatch_call_block_and_release + 12
10  libdispatch.dylib               0x000000011c906534 _dispatch_client_callout + 8
11  libdispatch.dylib               0x000000011c90c425 _dispatch_lane_serial_drain + 715
12  libdispatch.dylib               0x000000011c90cfa0 _dispatch_lane_invoke + 403
13  libdispatch.dylib               0x000000011c917591 _dispatch_workloop_worker_thread + 782
14  libsystem_pthread.dylib         0x000000011cdc8a3d _pthread_wqthread + 290
15  libsystem_pthread.dylib         0x000000011cdc7b77 start_wqthread + 15

Thread 7:: com.apple.uikit.eventfetch-thread
0   libsystem_kernel.dylib          0x000000011cd67dfa mach_msg_trap + 10
1   libsystem_kernel.dylib          0x000000011cd68170 mach_msg + 60
2   com.apple.CoreFoundation        0x0000000119ac8e77 __CFRunLoopServiceMachPort + 316
3   com.apple.CoreFoundation        0x0000000119ac358d __CFRunLoopRun + 1284
4   com.apple.CoreFoundation        0x0000000119ac2b9e CFRunLoopRunSpecific + 567
5   com.apple.Foundation            0x0000000114c20e61 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 209
6   com.apple.Foundation            0x0000000114c210d0 -[NSRunLoop(NSRunLoop) runUntilDate:] + 72
7   com.apple.UIKitCore             0x0000000123c7aa45 -[UIEventFetcher threadMain] + 464
8   com.apple.Foundation            0x0000000114c49521 __NSThread__start__ + 1042
9   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
10  libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 8:: com.apple.NSURLConnectionLoader
0   libsystem_kernel.dylib          0x000000011cd67dfa mach_msg_trap + 10
1   libsystem_kernel.dylib          0x000000011cd68170 mach_msg + 60
2   com.apple.CoreFoundation        0x0000000119ac8e77 __CFRunLoopServiceMachPort + 316
3   com.apple.CoreFoundation        0x0000000119ac358d __CFRunLoopRun + 1284
4   com.apple.CoreFoundation        0x0000000119ac2b9e CFRunLoopRunSpecific + 567
5   com.apple.CFNetwork             0x0000000112d7fb52 0x112b55000 + 2272082
6   com.apple.Foundation            0x0000000114c49521 __NSThread__start__ + 1042
7   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
8   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 9:
0   libsystem_pthread.dylib         0x000000011cdc7b68 start_wqthread + 0

Thread 10 Crashed:: Dispatch queue: com.apple.NSURLSession-delegate
0   com.apple.CoreFoundation        0x0000000119a863d6 CFDictionaryGetValue + 34
1   com.apple.Foundation            0x0000000114bad216 _NSSetLongLongValueAndNotify + 81
2   com.apple.CFNetwork             0x0000000112d1081e 0x112b55000 + 1816606
3   libdispatch.dylib               0x000000011c90535b _dispatch_call_block_and_release + 12
4   libdispatch.dylib               0x000000011c906534 _dispatch_client_callout + 8
5   libdispatch.dylib               0x000000011c90c425 _dispatch_lane_serial_drain + 715
6   libdispatch.dylib               0x000000011c90cfd7 _dispatch_lane_invoke + 458
7   libdispatch.dylib               0x000000011c917591 _dispatch_workloop_worker_thread + 782
8   libsystem_pthread.dylib         0x000000011cdc8a3d _pthread_wqthread + 290
9   libsystem_pthread.dylib         0x000000011cdc7b77 start_wqthread + 15

Thread 11:
0   libsystem_pthread.dylib         0x000000011cdc7b68 start_wqthread + 0

Thread 12:: com.google.firebase.crashlytics.MachExceptionServer
0   libsystem_kernel.dylib          0x000000011cd6abf6 write + 10
1   org.cocoapods.FirebaseCrashlytics   0x00000001149ea9de FIRCLSSDKFileLog + 406
2   org.cocoapods.FirebaseCrashlytics   0x00000001149ec813 FIRCLSMachExceptionServer + 1002 (FIRCLSMachException.c:170)
3   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
4   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 13:
0   libsystem_pthread.dylib         0x000000011cdc7b68 start_wqthread + 0

Thread 14:
0   libsystem_pthread.dylib         0x000000011cdc7b68 start_wqthread + 0

Thread 15:: AVAudioSession Notify Thread
0   libsystem_kernel.dylib          0x000000011cd67dfa mach_msg_trap + 10
1   libsystem_kernel.dylib          0x000000011cd68170 mach_msg + 60
2   com.apple.CoreFoundation        0x0000000119ac8e77 __CFRunLoopServiceMachPort + 316
3   com.apple.CoreFoundation        0x0000000119ac358d __CFRunLoopRun + 1284
4   com.apple.CoreFoundation        0x0000000119ac2b9e CFRunLoopRunSpecific + 567
5   com.apple.audio.AudioSession    0x000000012e259cde GenericRunLoopThread::Entry(void*) + 166
6   com.apple.audio.AudioSession    0x000000012e25b91b CAPThread::Entry(CAPThread*) + 77
7   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
8   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 16:: AMCP Logging Spool
0   libsystem_kernel.dylib          0x000000011cd67e36 semaphore_wait_trap + 10
1   com.apple.audio.caulk           0x0000000122c96ad2 caulk::mach::semaphore::wait_or_error() + 16
2   com.apple.audio.caulk           0x0000000122c8c5a6 caulk::semaphore::timed_wait(double) + 110
3   com.apple.audio.caulk           0x0000000122c90df4 caulk::concurrent::details::worker_thread::run() + 30
4   com.apple.audio.caulk           0x0000000122c90f50 void* caulk::thread_proxy<std::__1::tuple<caulk::thread::attributes, void (caulk::concurrent::details::worker_thread::*)(), std::__1::tuple<caulk::concurrent::details::worker_thread*> > >(void*) + 45
5   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
6   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 17:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 18:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 19:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 20:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 21:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 22:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 23:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 24:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 25:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 26:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 27:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 28:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 29:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 30:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 31:
0   libsystem_kernel.dylib          0x000000011cd67e4e semaphore_timedwait_trap + 10
1   libdispatch.dylib               0x000000011c906a4a _dispatch_sema4_timedwait + 76
2   libdispatch.dylib               0x000000011c906ea3 _dispatch_semaphore_wait_slow + 58
3   libdispatch.dylib               0x000000011c915eb2 _dispatch_worker_thread + 292
4   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
5   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 32:: com.apple.coreaudio.AQClient
0   libsystem_kernel.dylib          0x000000011cd67dfa mach_msg_trap + 10
1   libsystem_kernel.dylib          0x000000011cd68170 mach_msg + 60
2   com.apple.CoreFoundation        0x0000000119ac8e77 __CFRunLoopServiceMachPort + 316
3   com.apple.CoreFoundation        0x0000000119ac358d __CFRunLoopRun + 1284
4   com.apple.CoreFoundation        0x0000000119ac2b9e CFRunLoopRunSpecific + 567
5   com.apple.audio.toolbox.AudioToolbox    0x00000001195f6214 GenericRunLoopThread::Entry(void*) + 166
6   libAudioToolboxUtility.dylib    0x000000012ce975bd CADeprecated::CAPThread::Entry(CADeprecated::CAPThread*) + 77
7   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
8   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 33:
0   libsystem_kernel.dylib          0x000000011cd67e36 semaphore_wait_trap + 10
1   com.apple.audio.caulk           0x0000000122c96ad2 caulk::mach::semaphore::wait_or_error() + 16
2   com.apple.audio.caulk           0x0000000122c8c5a6 caulk::semaphore::timed_wait(double) + 110
3   com.apple.audio.caulk           0x0000000122c90df4 caulk::concurrent::details::worker_thread::run() + 30
4   com.apple.audio.caulk           0x0000000122c90f50 void* caulk::thread_proxy<std::__1::tuple<caulk::thread::attributes, void (caulk::concurrent::details::worker_thread::*)(), std::__1::tuple<caulk::concurrent::details::worker_thread*> > >(void*) + 45
5   libsystem_pthread.dylib         0x000000011cdcc109 _pthread_start + 148
6   libsystem_pthread.dylib         0x000000011cdc7b8b thread_start + 15

Thread 10 crashed with X86 Thread State (64-bit):
  rax: 0x000440370000001f  rbx: 0x0000000113039600  rcx: 0x0bf8dad4092c00d1  rdx: 0x0000000112fce10d
  rdi: 0x000440370000001f  rsi: 0x0000000112fce10d  rbp: 0x000070000824a7b0  rsp: 0x000070000824a780
   r8: 0x000070000824aa90   r9: 0x0000000000000001  r10: 0x0000600001510750  r11: 0x0000000114bad1c5
  r12: 0x0000000000000000  r13: 0x00007fb38cec5b30  r14: 0x0000000113039620  r15: 0x0000000112fce10d
  rip: 0x0000000119a863d6  rfl: 0x0000000000010202  cr2: 0x000000014ca23800

Logical CPU:     3
Error Code:      0x00000000
Trap Number:     13

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 555
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=959.9M resident=0K(0%) swapped_out_or_unallocated=959.9M(100%)
Writable regions: Total=620.7M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=620.7M(100%)

                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
Activity Tracing                   256K        1 
CG raster data                    2072K       59 
CoreAnimation                     1084K       65 
CoreServices                      3068K        1 
CoreUI image data                  452K        3 
Foundation                          16K        1 
IOSurface                         2472K        1 
Image IO                          2952K       11 
Kernel Alloc Once                    8K        1 
MALLOC                           197.9M       81 
MALLOC guard page                   48K       12 
MALLOC_NANO (reserved)           384.0M        1         reserved VM address space (unallocated)
Mach message                         4K        1 
SQLite page cache                  640K       10 
STACK GUARD                       56.1M       33 
Stack                             24.3M       35 
VM_ALLOCATE                       1136K       11 
__DATA                            32.9M      822 
__DATA_CONST                      48.4M      589 
__DATA_DIRTY                        16K        3 
__FONT_DATA                          4K        1 
__LINKEDIT                       380.0M      739 
__LLVM_COV                        1664K       32 
__TEXT                           580.5M      662 
__UNICODE                          588K        1 
libnetwork                         640K       16 
mapped file                      230.5M       20 
shared memory                       24K        3 
===========                     =======  ======= 
TOTAL                              1.9G     3215 
TOTAL, minus reserved VM space     1.5G     3215
paulb777 commented 3 years ago

Thanks @tspecht. I'll reopen. @visumickey, please take a look.

ergunkocak commented 3 years ago

Sorry for previous comment, my bad.

mthormann-dhi commented 3 years ago

@tspecht @paulb777 We're seeing this as well in our development CI builds on 7.0 as well as App Store version that released with 6.34.0 (pre-patch to Firebase Performance 3.3.2). We're hotfixing the App Store version to 6.34.0 post patch and will revert our dev code back from 7.0 till this is all resolved.

visumickey commented 3 years ago

@mthormann-dhi Since this is happening on the development CI builds, do you have any steps to reproduce this issue? Currently, we are working on all possibilities that could lead to such a crash and having a defined set of steps could expedite the fix greatly.

Can you also provide details about the APIs you are making network request with? Eg. dataTaskWithURLRequest, [NSURLSession sessionWithDelegate] and likes.

mthormann-dhi commented 3 years ago

We can't get it to happen on any particular test. We run approx. 80 UI tests and it seems to trigger on different ones each time. Test order is randomized as well though. Doesn't seem to happen when we run just one or two tests though but more the whole suite.

Most of our calls are done through Alamofire and/or AFNetworking. It's a mixed Objective-C/Swift project and we haven't been able to transition to just one.

I do have a source branch that can pretty reliably duplicate things running interactively as well in Xcode. I'll see if I can fix the test order on Monday and nail it down if that helps.

visumickey commented 3 years ago

@mthormann-dhi That would be greatly helpful. In the meantime, we will continue to cleanup all possible scenarios where this issue might occur.

Thanks for the information about Swift+ObjC app combined. We could add that to our mix of test cases.

Can you share the version of AlamoFire and AFNetworking you are using?

digitalby commented 3 years ago

I, too, started seeing random crashes like the ones described above after upgrading to version 7. Not using Alamofire though. The project is 100% Swift with SwiftUI and runs only on iOS 14+.

ethanwa commented 3 years ago

I am also seeing 1000's of crashes per day from my users with 7.0 (update went out 2 days ago). I have to revert my app back with an emergency update, and customers are pissed. The crashes happen randomly within the app and there are no set repro steps. Sometimes it happens when just opening views or pressing buttons that have no network calls associated with them. I am using Alamofire 5.3.0. Here's the trace:

Screen Shot 2020-10-31 at 4 57 09 PM
mthormann-dhi commented 3 years ago

Alamofire 5.2.2 AFNetworking 4.0.1

Like the other comments we're seeing errors all over looks like related to KVO type stuff. Some stack traces as well.

Crashed: org.alamofire.session.rootQueue 0 libobjc.A.dylib 0x1c0f8e010 objc_retain + 16 1 Foundation 0x1ae1ea8f0 NSKeyValueWillChange + 404 2 Foundation 0x1ae1ea5b8 -[NSObject(NSKeyValueObservingPrivate) _changeValueForKeys:count:maybeOldValuesDict:maybeNewValuesDict:usingBlock:] + 432 3 Foundation 0x1ae12711c -[NSObject(NSKeyValueObservingPrivate) _changeValueForKey:key:key:usingBlock:] + 72 4 Foundation 0x1ae1e573c _NSSetLongLongValueAndNotify + 312 5 CFNetwork 0x1ad6b54c4 _CFNetworkHTTPConnectionCacheSetLimit + 150756 6 Foundation 0x1ae20e7f4 __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 24 7 Foundation 0x1ae101120 -[NSBlockOperation main] + 104 8 Foundation 0x1ae210bb8 __NSOPERATION_IS_INVOKING_MAIN__ + 24 9 Foundation 0x1ae100da8 -[NSOperation start] + 808 10 Foundation 0x1ae211624 __NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__ + 24 11 Foundation 0x1ae2110d0 __NSOQSchedule_f + 184 12 libdispatch.dylib 0x1acb02834 _dispatch_block_async_invoke2 + 148 13 libdispatch.dylib 0x1acaf4ac8 _dispatch_client_callout + 20 14 libdispatch.dylib 0x1acafbc08 _dispatch_lane_serial_drain + 580 15 libdispatch.dylib 0x1acafc734 _dispatch_lane_invoke + 408 16 libdispatch.dylib 0x1acb06528 _dispatch_workloop_worker_thread + 708 17 libsystem_pthread.dylib 0x1f4448908 _pthread_wqthread + 276 18 libsystem_pthread.dylib 0x1f444f77c start_wqthread + 8

Crashed: com.apple.main-thread 0 CoreFoundation 0x1a563d35c CFDictionaryGetValue + 88 1 Foundation 0x1a69e9660 _NSSetLongLongValueAndNotify + 92 2 libdispatch.dylib 0x1a52f2fd0 _dispatch_call_block_and_release + 32 3 libdispatch.dylib 0x1a52f4ac8 _dispatch_client_callout + 20 4 libdispatch.dylib 0x1a530204c _dispatch_main_queue_callback_4CF + 836 5 CoreFoundation 0x1a56731e4 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16 6 CoreFoundation 0x1a566d3b4 __CFRunLoopRun + 2508 7 CoreFoundation 0x1a566c4bc CFRunLoopRunSpecific + 600 8 GraphicsServices 0x1bc17e820 GSEventRunModal + 164 9 UIKitCore 0x1a8019164 -[UIApplication _run] + 1072 10 UIKitCore 0x1a801e840 UIApplicationMain + 168 11 Dice 0x104367690 main + 9 (main.swift:9) 12 libdyld.dylib 0x1a5333e40 start + 4

Crashed: com.apple.NSURLSession-work 0 libobjc.A.dylib 0x1b6761dc0 objc_release + 16 1 Foundation 0x1b6df9a24 NSKeyValuePopPendingNotificationLocal + 152 2 Foundation 0x1b6df9944 NSKeyValueDidChange + 372 3 Foundation 0x1b6df9398 -[NSObject(NSKeyValueObservingPrivate) _changeValueForKeys:count:maybeOldValuesDict:maybeNewValuesDict:usingBlock:] + 632 4 Foundation 0x1b6d42f60 -[NSObject(NSKeyValueObservingPrivate) _changeValueForKey:key:key:usingBlock:] + 72 5 Foundation 0x1b6df4d88 _NSSetLongLongValueAndNotify + 300 6 CFNetwork 0x1b9cb95ec CFNetServiceBrowserSearchForServices + 1784 7 CFNetwork 0x1b9edba8c __CFTubeSetTubeTypeNotifier + 42716 8 libdispatch.dylib 0x1b66d2134 _dispatch_call_block_and_release + 32 9 libdispatch.dylib 0x1b66d35ac _dispatch_client_callout + 20 10 libdispatch.dylib 0x1b66d9a64 _dispatch_lane_serial_drain + 568 11 libdispatch.dylib 0x1b66da4cc _dispatch_lane_invoke + 452 12 libdispatch.dylib 0x1b66e3a5c _dispatch_workloop_worker_thread + 584 13 libsystem_pthread.dylib 0x1b6739718 _pthread_wqthread + 276 14 libsystem_pthread.dylib 0x1b673f9c8 start_wqthread + 8

Crashed: com.sift.SFUploader 0 libobjc.A.dylib 0x1b7c02160 objc_release + 16 1 Foundation 0x1a4deed18 NSKeyValuePopPendingNotificationLocal + 152 2 Foundation 0x1a4deec38 NSKeyValueDidChange + 360 3 Foundation 0x1a4dee690 -[NSObject(NSKeyValueObservingPrivate) _changeValueForKeys:count:maybeOldValuesDict:maybeNewValuesDict:usingBlock:] + 648 4 Foundation 0x1a4d2b11c -[NSObject(NSKeyValueObservingPrivate) _changeValueForKey:key:key:usingBlock:] + 72 5 Foundation 0x1a4de973c _NSSetLongLongValueAndNotify + 312 6 CFNetwork 0x1a433f3b0 __CFTubeSetTubeTypeNotifier + 57112 7 AFNetworking 0x10095d8a8 -[_AFURLSessionTaskSwizzling af_resume] + 430 (AFURLSessionManager.m:430) 8 Sift 0x10113b2d8 __24-[SiftUploader doUpload]_block_invoke_2 + 192 (SiftUploader.m:192) 9 libdispatch.dylib 0x1a36f2fd0 _dispatch_call_block_and_release + 32 10 libdispatch.dylib 0x1a36f4ac8 _dispatch_client_callout + 20 11 libdispatch.dylib 0x1a36fbc08 _dispatch_lane_serial_drain + 580 12 libdispatch.dylib 0x1a36fc734 _dispatch_lane_invoke + 408 13 libdispatch.dylib 0x1a3706528 _dispatch_workloop_worker_thread + 708 14 libsystem_pthread.dylib 0x1eb9e7908 _pthread_wqthread + 276 15 libsystem_pthread.dylib 0x1eb9ee77c start_wqthread + 8

theofficejackpot commented 3 years ago

I'll chime in here as well. I have finally found a scenario in my app that can crash every time (Crashed: org.alamofire.session.rootQueue). I had Alamofire 4.8 (legacy) and thought that was the culprit because it was crashing on an Alamofire thread, so I spent a LOT of time over this weekend updating to Alamofire 5.3. Still crashed.
Then I came across this thread.

I can confirm, zero crashes after downgrading to 6.34. See my pod install downgrade log:

Analyzing dependencies Downloading dependencies Installing Firebase 6.34.0 (was 7.0.0) Installing FirebaseABTesting 4.2.0 (was 7.0.0) Installing FirebaseAnalytics 6.9.0 (was 7.0.0) Installing FirebaseAuth 6.9.2 (was 7.0.0) Installing FirebaseCore 6.10.4 (was 7.0.0) Installing FirebaseCoreDiagnostics 1.7.0 (was 7.0.0) Installing FirebaseCrashlytics 4.6.2 (was 7.0.0) Installing FirebaseInstallations 1.7.0 (was 7.0.0) Installing FirebaseInstanceID 4.8.0 (was 7.0.0) Installing FirebaseMessaging 4.7.1 (was 7.0.0) Installing FirebasePerformance 3.3.2 (was 7.0.0) Installing FirebaseRemoteConfig 4.9.1 (was 7.0.0) Installing Google-Mobile-Ads-SDK 7.67.0 (was 7.67.1) Installing GoogleAppMeasurement 6.9.0 (was 7.0.0) Installing GoogleDataTransport 7.5.1 (was 8.0.0) Installing GoogleUtilities 6.7.2 (was 7.0.0) Installing nanopb 1.30906.0 (was 2.30906.0) Generating Pods project Integrating client project

I was able to do this by setting this in my pod file: pod 'Firebase/Core', '~> 6.33'

Then running these commands: rm -rf Podfile.lock Pods/;pod install --repo-update

Then cleaning build folder in Xcode, letting it index, then building.

visumickey commented 3 years ago

Thank a lot for such useful stack traces. After a lot of reads, I think this crash is related to NSURLSession not compatible with KVO on the property 'state'. Recently Firebase Performance started observing on the 'state' property to identify the completion of the network request and that seems error prone (This was required because of iOS14).

Reference to similar kind of crash: https://github.com/AFNetworking/AFNetworking/issues/1477, https://github.com/pinterest/PINRemoteImage/issues/390

To mitigate this, we will remove the KVO based implementation with an alternate mechanism to stop these crashes from occurring.

petrykDima commented 3 years ago

@maksymmalyhin which solution is best right now? downgrading to 6.34 and waiting for new Firebase release?

petrykDima commented 3 years ago

@visumickey maybe you can recommend something in current situation. I have about 800 crashes after release new version in one day =)

https://github.com/firebase/firebase-ios-sdk/issues/6734#issuecomment-721188753

paulb777 commented 3 years ago

Two workarounds until the crash is addressed in Firebase 7.x:

petrykDima commented 3 years ago

@paulb777 thanks

tspecht commented 3 years ago

@petrykDima we have removed everything from the Firebase suite that was not 💯 necessary and will not be back any time soon ...

visumickey commented 3 years ago

We have released Firebase performance version 7.0.1 that removes the KVO implementation. Please update to the recent version of Firebase Performance to get away from the crashes users are experiencing.

Thanks a lot of staying up and providing details information that enabled us to know about the issue with the Apple framework.

tspecht commented 3 years ago

@visumickey was a new Firebase version released for this in general? Just to be clear, we stopped using Firebase/Performance a couple of weeks ago when these issues first started happening, but even updating the remaining FIrebase pods to 7.0.0 still had those issues. Here's the list of what we currently have installed on 7.0.0

  pod "Firebase/Crashlytics", FirebaseVersion
  pod "Firebase/RemoteConfig", FirebaseVersion
  pod "Firebase/Analytics", FirebaseVersion
  pod "Firebase/Auth", FirebaseVersion
  pod "Firebase/AppDistribution", FirebaseVersion
visumickey commented 3 years ago

@tspecht This emergency fix has only Firebase performance in it. You can set only the Firebase performance SDK's version to 7.0.1 to get past the issue. Other pods can continue to use the same version of Firebase you have.

visumickey commented 3 years ago

That is surprising. Based on the stack trace provided above, these issues were arising out of Firebase performance. If you are seeing this still, quite likely it is because of your other dependencies (most likely non-Firebase dependencies).