Closed fangyuxi closed 5 years ago
@paulb777 I am still using XCode 10.2.1 for distributing my app.
Here is more information about our crash, Hope this will help Firebase SDK team here
Podfile:
pod 'Firebase/Core', '6.2.0'
pod 'Firebase/DynamicLinks', '6.2.0'
pod 'Firebase/Performance', '6.2.0'
pod 'Firebase/Messaging', '6.2.0'
Podfile.lock
- Firebase/Core (6.2.0):
- Firebase/CoreOnly
- FirebaseAnalytics (= 6.0.1)
- Firebase/CoreOnly (6.2.0):
- FirebaseCore (= 6.0.2)
- Firebase/DynamicLinks (6.2.0):
- Firebase/CoreOnly
- FirebaseDynamicLinks (~> 4.0.0)
- Firebase/Messaging (6.2.0):
- Firebase/CoreOnly
- FirebaseMessaging (~> 4.0.2)
- Firebase/Performance (6.2.0):
- Firebase/CoreOnly
- FirebasePerformance (~> 3.0.0)
- FirebaseABTesting (3.0.0):
- FirebaseCore (~> 6.0)
- Protobuf (~> 3.5)
- FirebaseAnalytics (6.0.1):
- FirebaseCore (~> 6.0)
- FirebaseInstanceID (~> 4.1)
- GoogleAppMeasurement (= 6.0.1)
- GoogleUtilities/AppDelegateSwizzler (~> 6.0)
- GoogleUtilities/MethodSwizzler (~> 6.0)
- GoogleUtilities/Network (~> 6.0)
- "GoogleUtilities/NSData+zlib (~> 6.0)"
- nanopb (~> 0.3)
- FirebaseAnalyticsInterop (1.2.0)
- FirebaseCore (6.0.2):
- GoogleUtilities/Environment (~> 6.0)
- GoogleUtilities/Logger (~> 6.0)
- FirebaseDynamicLinks (4.0.0):
- FirebaseAnalyticsInterop (~> 1.0)
- FirebaseCore (~> 6.0)
- FirebaseInstanceID (4.1.1):
- FirebaseCore (~> 6.0)
- GoogleUtilities/Environment (~> 6.0)
- GoogleUtilities/UserDefaults (~> 6.0)
- FirebaseMessaging (4.0.2):
- FirebaseAnalyticsInterop (~> 1.1)
- FirebaseCore (~> 6.0)
- FirebaseInstanceID (~> 4.1)
- GoogleUtilities/AppDelegateSwizzler (~> 6.2)
- GoogleUtilities/Environment (~> 6.2)
- GoogleUtilities/Reachability (~> 6.2)
- GoogleUtilities/UserDefaults (~> 6.2)
- Protobuf (~> 3.1)
- FirebasePerformance (3.0.0):
- FirebaseCore (~> 6.0)
- FirebaseInstanceID (~> 4.0)
- FirebaseRemoteConfig (~> 4.0)
- GoogleToolboxForMac/Logger (~> 2.1)
- "GoogleToolboxForMac/NSData+zlib (~> 2.1)"
- GoogleUtilities/Environment (~> 6.0)
- GoogleUtilities/ISASwizzler (~> 6.0)
- GoogleUtilities/MethodSwizzler (~> 6.0)
- GTMSessionFetcher/Core (~> 1.1)
- Protobuf (~> 3.5)
- FirebaseRemoteConfig (4.0.0):
- FirebaseABTesting (~> 3.0)
- FirebaseCore (~> 6.0)
- FirebaseInstanceID (~> 4.0)
- GoogleUtilities/Environment (~> 6.0)
- "GoogleUtilities/NSData+zlib (~> 6.0)"
- Protobuf (~> 3.5)
Fabric info:
Hope this will help. @paulb777 @visumickey
Hi, we also face this issue. for iOS10 users
- Firebase/Analytics (6.6.0):
- Firebase/Core
- Firebase/Core (6.6.0):
- Firebase/CoreOnly
- FirebaseAnalytics (= 6.1.0)
- Firebase/CoreOnly (6.6.0):
- FirebaseCore (= 6.2.0)
- Firebase/Performance (6.6.0):
- Firebase/CoreOnly
- FirebasePerformance (~> 3.1.2)
- FirebaseABTesting (3.0.0):
- FirebaseCore (~> 6.0)
- Protobuf (~> 3.5)
- FirebaseAnalytics (6.1.0):
- FirebaseCore (~> 6.2)
- FirebaseInstanceID (~> 4.2)
- GoogleAppMeasurement (= 6.1.0)
- GoogleUtilities/AppDelegateSwizzler (~> 6.0)
- GoogleUtilities/MethodSwizzler (~> 6.0)
- GoogleUtilities/Network (~> 6.0)
- "GoogleUtilities/NSData+zlib (~> 6.0)"
- nanopb (~> 0.3)
- FirebaseCore (6.2.0):
- FirebaseCoreDiagnostics (~> 1.0)
- FirebaseCoreDiagnosticsInterop (~> 1.0)
- GoogleUtilities/Environment (~> 6.2)
- GoogleUtilities/Logger (~> 6.2)
- FirebaseCoreDiagnostics (1.0.1):
- FirebaseCoreDiagnosticsInterop (~> 1.0)
- GoogleDataTransportCCTSupport (~> 1.0)
- GoogleUtilities/Environment (~> 6.2)
- GoogleUtilities/Logger (~> 6.2)
- FirebaseCoreDiagnosticsInterop (1.0.0)
- FirebaseInstanceID (4.2.3):
- FirebaseCore (~> 6.0)
- GoogleUtilities/Environment (~> 6.0)
- GoogleUtilities/UserDefaults (~> 6.0)
- FirebasePerformance (3.1.2):
- FirebaseCore (~> 6.2)
- FirebaseInstanceID (~> 4.2)
- FirebaseRemoteConfig (~> 4.2)
- GoogleToolboxForMac/Logger (~> 2.1)
- "GoogleToolboxForMac/NSData+zlib (~> 2.1)"
- GoogleUtilities/Environment (~> 6.0)
- GoogleUtilities/ISASwizzler (~> 6.0)
- GoogleUtilities/MethodSwizzler (~> 6.0)
- GTMSessionFetcher/Core (~> 1.1)
- Protobuf (~> 3.5)
- FirebaseRemoteConfig (4.2.1):
- FirebaseABTesting (~> 3.0)
- FirebaseCore (~> 6.1)
- FirebaseInstanceID (~> 4.2)
- GoogleUtilities/Environment (~> 6.0)
- "GoogleUtilities/NSData+zlib (~> 6.0)"
- Protobuf (~> 3.5)
...
- Protobuf (3.9.0)
...
the crash stack is
and every crash log, it also has another thread call stack about CCTLogWriter
(not all of the crash log is the same stack.)
I'm not sure what relative between these two stack. Hope can help this issue
@ChristianVinterly Yours is a Firestore/Protobuf issue, not related to Phenotype. Please file a separate issue.
Thanks a lot @patrickmarshall @Strate @willsbor for sharing the crash details. Thanks everyone for chiming:
Trying to get some details clarified:
Based on our understanding until now, we do not expect this to be an SDK issue. We are still working hard to figure out what could have caused this impact.
Please share your inputs on the questions above. That can help us great!
@visumickey
I've tried to downgrade Firebase dependencies to ~> 5.20.1
, build new version to Test Flight and asked one of my users to test (I do not have iOS 10 device) - it still fails, with exactly same stack trace & error. Btw, previous versions of my app, which are uses exactly same version of Pod are not affected - I do not see any errors in Crashlytics from previous app builds.
Can't answer, because my app released later
Volume of crashes is increased by 100% right now (our new version in production for 2 days). Looks like all iOS9 & ios10 devices are affected.
Top of my affected devices are iPhone 5 & iPhone 5c
Now I check the theory that this is build issue. When I test on iPhone 5 & ios10 simulator via "Play" button in Xcode everything is fine (no crash). Now I want to archive and export .ipa-file to test on emulator.
Thanks @Strate. Can you share the podfile/podfile.lock after downgrading firebase to 5.20.1? May be we can try reproducing with that combination.
@visumickey
Thanks @Strate. Will try reproducing with this combination.
@visumickey
- Any instance of these crashes happening only on the latest build of the App (After updating to the latest version of Firebase)? (We are trying to ascertain if this is occurring only on the latest version. Our understanding based on the comments above is NO).
after the version which we face this issue, we release a new version. but we doesn't change the Firebase version. So we still see the crash report on crashlytics. and we do not see any errors in Crashlytics from previous app builds.
- Have the crashes started showing only from Sep 13th? (Trying to narrow down the date range)
our time zone is +8:00
- Has the volume of crashes slowed down or is that still in the same range? (Trying to make sure if this is not sporadic).
I'm not sure.
- Apart from iOS 9 and iOS 10, do we have any insights on the kind of devices where this issue is more dominant? (Helps us to reproduce the issue).
Thanks 🙏
@morganchen12 Sure, I can create another issue. But it has the exact same stack trace in Protobuf, even if call site from your framework is different.
@visumickey
Another information...
in Sep 12, we released a version 2.36.1, but it had a crash issue (not this). so we fixed it and released a version 2.36.2 in Sep 13. This version just fix a bug without changing Podfile / Podfile.lock.
the result is "2.36.1 didn't face this crash issue, but 2.36.2 has this crash." (These two versions are separated by 1 to 2 days.)
I'm not sure what happened between the fix and Firebase. But that fix is modifying object creating timing. 🙏
Thanks for sharing the device information.
Given the device information we've seen so far, it seems that this is a 32-bit device only crash. Has anyone seen this on a 64-bit device?
The crash occurs converting a stream to a double, so it could be an unaligned access error. It's still totally a mystery to me why this just started occurring recently.
I've reproduced the crash on an iPhone 5c with the Firestore quickstart app. It crashes every time when building in release mode.
I found a workaround:
Update the GPBConvertInt64ToDouble function in . ./Pods/Protobuf/objectivec/GPBUtilities_PackagePrivate.h
as follows with the volatile
:
GPB_INLINE double GPBConvertInt64ToDouble(int64_t v) { volatile union { double f; int64_t i; } u; u.i = v; return u.f; }
Here's my best guess rationale:
There's an optimizer bug such that the compiler does not properly honor the alignment restrictions of an inline function that can result in an unaligned exception when accessing 64 bit data on a 32 bit processor.
I suspect the bug has been there a while, but has been exposed by a recent backend protocol buffers change.
We'll follow up with the Protocol Buffers team on a fix.
And here's a workaround that won't impact 64-bit performance:
GPB_INLINE double GPBConvertInt64ToDouble(int64_t v) {
#if __LP64__
union { double f; int64_t i; } u;
#else
volatile union { double f; int64_t i; } u;
#endif
u.i = v;
return u.f;
}
@paulb777 That is great that you managed to reproduce and resolve. Are you releasing an urgent update? and if so, when do you expect? (Will it be an emergency release?) Or are we better to patch with this ourselves? Thanks.
@neilmorton The patch timeline is still TBD. Since the crash is in a Firebase dependency and not in Firebase itself, we need to coordinate the fix and release with the Protobuf team. Also, we don't yet understand what made this bug all of a sudden occur starting last week.
It would also be helpful if others could reproduce with a release configuration build running on a 32 bit device and then verify the workaround.
@paulb777 Thanks for the clarification.
Unfortunately I haven't got anyone I can reach out to with a 32 bit device.
How confident are you in this fix? (Only asking before we start adding it manually!). Thanks.
I'm confident adding the volatile
fixes the crash in the Firestore quickstart and it seems likely that it should fix any similar crashes.
However, given that we don't yet fully understand the root causes, there is still some uncertainty.
In summary, I would recommend that anyone using FirebaseFirestore or FirebasePerformance who cares about 32-bit device support to ship an update with the patch, ideally after verification if you can access a 32-bit device.
The patch should be a no-op for 64-bit devices and fix a likely crash for 32-bit devices.
@paulb777 is you release ipa have a bitcode option YES, I guess it is the bitcode option cause, I have the same issue, and I have not updated the SDK.
@fangyuxi did you release your app with a bitcode option?
@freezy7 YES
@fangyuxi I think it is caused by uploading ipa to App Connect and choosing rebuild with bicode. I think I can try not to select this option first, but it takes a certain amount of time to submit the review. I have not tried it yet.
@freezy7 可能不是bitcode的原因。还是别试了。
@paulb777 thank you for workaround. I confirm that this fixes crash on iPhone 5 device for my app.
@Strate Hello, Do I need to update my app or Firebase to fix this Crash ?
@fagyuxi I applied workaround manually and built new version of app to AppStore
We're continuing to investigate. The initial proposed Protobuf PR (see above) still causes an alignment exception. The best known workaround is still to use the volatile
change documented above.
Also, even though the initial bug report says Xcode 10 was used, we're only able to reproduce with Xcode 11 builds. Is anyone else seeing the issue from an Xcode 10 build?
@paulb777 I saw one instance in an app which was dated around the 8th and was on an Xcode 10 build. But looking now I can't see it in Fabric. So, I think it may have been an anomaly. The rest of mine were with Xcode 11.
@paulb777 I use xCode 10 right now. Can confirm crash & workaround.
Yes, we are getting this issue with Xcode 10.2.1 also.
We are getting same crash on iOS 11 and 12 with 64 Bit Devices.
Our production build that crashes is built with Xcode 10.3. We don't have any builds in production with Xcode 11 yet.
We are getting same crash on iOS 11 and 12 with 64 Bit Devices.
@Nishant-goibibo what specific device? I didn't think arm64 had unaligned memory issues. Or are you maybe build 32bit only and running that on arm64?
The protobuf fix is in progress at https://github.com/protocolbuffers/protobuf/pull/6678
hey everyone! when will be update?
Protobuf 3.9.2 has now been published to CocoaPods with the fix.
pod update
to pick it up.
Thanks to everyone who contributed to this thread with backtraces and detailed crash information that helped us track this one down!
@paulb777 any timeline for a carthage update including the fix?
@datalogen Are you seeing the crash with a Carthage distribution?
@paulb777 Yes, we are seeing a similar crash with a Carthage distribution. We have quite a few customers running on older devices - iOS 9 and 10.
Crashed: com.google.firebase.firestore
0 DomainServices 0x102f91c GPBCodedInputStreamReadDouble + 19
1 DomainServices 0x102f915 GPBCodedInputStreamReadDouble + 12
2 DomainServices 0x105b135 -[GPBMessage mergeFromCodedInputStream:extensionRegistry:] + 9112
3 DomainServices 0x1030315 -[GPBCodedInputStream readMessage:extensionRegistry:] + 1406
4 DomainServices 0x1036e33 ReadValue + 1148
5 DomainServices 0x1036a93 GPBDictionaryReadEntry + 220
6 DomainServices 0x103039d -[GPBCodedInputStream readMapEntry:extensionRegistry:field:parentMessage:] + 1542
7 DomainServices 0x105b0ab -[GPBMessage mergeFromCodedInputStream:extensionRegistry:] + 8974
8 DomainServices 0x1030315 -[GPBCodedInputStream readMessage:extensionRegistry:] + 1406
9 DomainServices 0x105b2b3 -[GPBMessage mergeFromCodedInputStream:extensionRegistry:] + 9494
10 DomainServices 0x105a7ab -[GPBMessage mergeFromData:extensionRegistry:] + 6670
11 DomainServices 0x1057e99 -[GPBMessage initWithData:extensionRegistry:error:] + 768
12 DomainServices 0x105a87f +[GPBMessage parseFromData:extensionRegistry:error:] + 6882
13 DomainServices 0x105a849 +[GPBMessage parseFromData:error:] + 6828
14 DomainServices 0xcb2669 firebase::firestore::local::LevelDbRemoteDocumentCache::DecodeMaybeDocument(absl::string_view, firebase::firestore::model::DocumentKey const&) + 200
15 DomainServices 0xcb2fcf firebase::firestore::local::LevelDbRemoteDocumentCache::GetMatching(firebase::firestore::core::Query const&) + 370
16 DomainServices 0xcb6e55 firebase::firestore::local::LocalDocumentsView::GetDocumentsMatchingCollectionQuery(firebase::firestore::core::Query const&) + 44
17 DomainServices 0xcb65b1 firebase::firestore::local::LocalDocumentsView::GetDocumentsMatchingQuery(firebase::firestore::core::Query const&) + 56
18 DomainServices 0xc6bd5d -[FSTLocalStore executeQuery:] + 115832
19 DomainServices 0xc8a6ef -[FSTSyncEngine initializeViewAndComputeSnapshotForQueryData:] + 241162
20 DomainServices 0xc89ffd -[FSTSyncEngine listenToQuery:] + 239384
21 DomainServices 0xc1baef firebase::firestore::core::EventManager::AddQueryListener(std::1::shared_ptr
@paulb777 Debug builds are not showing the issue - release builds however are consistently crashing on my 32-bit device running iOS 9.3.5.
Thanks @datalogen. The fix will be included in the Firebase 6.9.0 Carthage (and zip) release that is planned to go out early in the week.
Hello, We have release our apps since september 27th and the problem doesn't seems to be appear anymore. We just want to say thanks to @paulb777 @visumickey and other google team that help to fixed this issue.
I'm using firebase 6.8.1 When new App is released at Appstore, I found more crashs appear in firebase dashboard. I do not use ProtoBuf directly I updated all Pods, before upload package to Appstore. All componets are newest.
This is the details of the App that has Crash.
The pod install info.