Closed friedbunny closed 5 years ago
I wanted to clarify that, in the Symbolicated / Unsymbolicated examples above, the Symbolicated log is also only partially symbolicated:
1 Mapbox 0x0000000100409734 mbgl::DefaultFileSource::resume() + 24
2 Mapbox 0x0000000100409734 mbgl::DefaultFileSource::resume() + 24
3 Mapbox 0x00000001003c9c1c _hidden#8730_ + 24 (__hidden#8849_:59)
The "_hidden" symbol in this particular case would've been:
3 Mapbox 0x00000001003c9c1c -[MGLOfflineStorage unpauseFileSource:] (MGLOfflineStorage.mm:59)
@friedbunny
It appears that this is a bug in more recent versions of Xcode and a side affect of Bitcodeification. A very similar issue was reported in this open radar.
I downloaded the dSYM file from iTC for the first Symbolicated (that was actually partially symbolicated) crash report noted above and performed the repro steps noted in that radar. I confirmed that the dSYM Apple sends back contains many "_hidden" symbols:
nm CC99108F-9E76-39FE-B15F-D105D8406611.dSYM/Contents/Resources/DWARF/* | grep _hidden
0000000000005d98 t __hidden#0_
00000000000fada4 t __hidden#10000_
00000000000fada8 t __hidden#10001_
00000000000fae40 t __hidden#10002_
...
Happily, the Open Radar report was updated with a response from Apple Developer Relations and it does look like there is a chance this will be fixed in Xcode 8.3. I don't think there is much we can do at the moment other than hope for the best with Xcode 8.3 and also file a duplicate of https://openradar.appspot.com/28503166. I think we should move this issue of the 3.5.1 milestone but keep it open for our own tracking purposes. I'll update this ticket with the Open Radar dupe once I file it.
Sounds reasonable, @boundsj. It seems like Apple’s track record with bitcode and dSYMs isn’t great, judging by these Crashlytics users' experiences.
We are publicly tracking in a duplicate Open Radar.
Still seeing this (via Crashlytics) in a TestFlight app built with Xcode 8.3.
https://openradar.appspot.com/28503166 was closed as resolved in Xcode 8.3, but inspection of dSYMs downloaded from TestFlight don’t show any significant differences between those built with 8.2 or 8.3.
We’ll keep an eye on this issue and close when crash logs are consistently usefully symbolicated. In the meantime, v3.5.2 will be built with Xcode 8.3.1.
I seem to have the same problem, using the iOS SDK version 3.7.3 I get a crash that appears frequently in my app, and I can't seem to get more info than this:
0 Mapbox 0x102f17da0 _hidden#13907_ + 379760
1 Mapbox 0x102f182fc _hidden#13908_ + 381132
2 Mapbox 0x102f69408 _hidden#15669_ + 713176
3 Mapbox 0x102f7bf20 _hidden#15927_ + 789744
4 Mapbox 0x102e44c2c _hidden#7149_ (__hidden#8198_:53)
5 Mapbox 0x102e44bd0 _hidden#7148_ (__hidden#8038_:950)
6 GLKit 0x18f65492c -[GLKView _display:] + 216
7 Mapbox 0x102e453dc _hidden#7157_ (__hidden#8038_:1081)
8 QuartzCore 0x1866af710 CA::Display::DisplayLink::dispatch_items(unsigned long long, unsigned long long, unsigned long long) + 672
9 IOKit 0x1829ed098 IODispatchCalloutFromCFMessage + 392
10 CoreFoundation 0x182710290 __CFMachPortPerform + 188
11 CoreFoundation 0x18272b000 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 56
12 CoreFoundation 0x18272a704 __CFRunLoopDoSource1 + 440
13 CoreFoundation 0x1827281d8 __CFRunLoopRun + 2196
14 CoreFoundation 0x182647e58 CFRunLoopRunSpecific + 436
15 GraphicsServices 0x1844f4f84 GSEventRunModal + 100
16 UIKit 0x18bd9c67c UIApplicationMain + 236```
We’ve been seeing a related issue for some time (since around ios-v3.7.0-alpha.1
in September 2017) where symbols from our core are consistently replaced with utrie2_enumForLeadSurrogate_58
. This can apparently manifest on any crash, but we haven’t identified a cause or set of reproduction circumstances yet.
For example, here’s one such crash that ended up being #11940 (and ultimately ended up with a very different stack trace):
Assertion failed: (!expression::isZoomConstant(*expression)), function CameraFunction, file /Users/distiller/project/include/mbgl/style/function/camera_function.hpp, line 34.
Crashed: com.apple.main-thread
0 libsystem_kernel.dylib 0x183d792ec __pthread_kill + 8
1 libsystem_pthread.dylib 0x183f1e6a8 pthread_kill$VARIANT$armv81 + 360
2 libsystem_c.dylib 0x183ce7d0c abort + 140
3 libsystem_c.dylib 0x183cbc000 basename_r + 314
4 Mapbox 0x1003e1140 MGLTimeIntervalFromDuration(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> >) + 122684
5 Mapbox 0x1003e09d8 MGLTimeIntervalFromDuration(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> >) + 120788
6 Mapbox 0x1004f11f4 utrie2_enumForLeadSurrogate_58 + 1016688
7 Mapbox 0x1004ded5c utrie2_enumForLeadSurrogate_58 + 941784
8 Mapbox 0x1004dea64 utrie2_enumForLeadSurrogate_58 + 941024
9 Mapbox 0x1004e4590 utrie2_enumForLeadSurrogate_58 + 964364
10 Mapbox 0x1004e531c utrie2_enumForLeadSurrogate_58 + 967832
11 Mapbox 0x10057289c utrie2_enumForLeadSurrogate_58 + 1546776
12 Mapbox 0x1005724f0 utrie2_enumForLeadSurrogate_58 + 1545836
13 Mapbox 0x100571ae0 utrie2_enumForLeadSurrogate_58 + 1543260
14 Mapbox 0x100570ad0 utrie2_enumForLeadSurrogate_58 + 1539148
15 Mapbox 0x10057de24 utrie2_enumForLeadSurrogate_58 + 1593248
16 Mapbox 0x1005800c4 utrie2_enumForLeadSurrogate_58 + 1602112
17 Mapbox 0x1005e1990 utrie2_enumForLeadSurrogate_58 + 2001676
18 Mapbox 0x100403854 utrie2_enumForLeadSurrogate_58 + 43472
19 Mapbox 0x1004039e8 utrie2_enumForLeadSurrogate_58 + 43876
20 Mapbox 0x1005d4810 utrie2_enumForLeadSurrogate_58 + 1948044
21 Mapbox 0x1005d3cec utrie2_enumForLeadSurrogate_58 + 1945192
22 CoreFoundation 0x18429b404 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 24
23 CoreFoundation 0x18429ac2c __CFRunLoopDoSources0 + 276
24 CoreFoundation 0x18429879c __CFRunLoopRun + 1204
25 CoreFoundation 0x1841b8da8 CFRunLoopRunSpecific + 552
26 GraphicsServices 0x18619b020 GSEventRunModal + 100
27 UIKit 0x18e19978c UIApplicationMain + 236
28 StudioPreview 0x100077d14 main (OfflineStorage.swift:13)
29 libdyld.dylib 0x183c49fc0 start + 4
We’ve been seeing a related issue for some time (since around
ios-v3.7.0-alpha.1
in September 2017) where symbols from our core are consistent replaced withutrie2_enumForLeadSurrogate_58
.
Stabbing in the dark here, but could this be an unrelated issue caused by -fvisibility=hidden
? #7604 landed in both v3.6.1 and ios-v3.7.0-alpha.1. What puzzles me is that utrie2_enumForLeadSurrogate_58
would’ve been linked in from libicuuc via mbgl, and libicuuc is built with -fvisibility=hidden
.
000000000013ad47 T _utrie2_enumForLeadSurrogate_58
can be found in our release dSYMs (as of 4.1.0b1), as can all of the expected mbgl
symbols. The utrie
symbols are almost the final ones in the dSYM (at least, if nm
’s ordering is to be believed), so...?
Both of these issues (the original _hidden
and the new _utrie
) appear to stem from bitcode and the lack of .bcsymbolmap
files in the submitted .xcarchive
— manually adding our framework’s . bcsymbolmap
s to a test app’s product folder (via a copy files build phase) was enough to get properly symbolicated core crashes in Crashlytics.
Symptoms I observed:
_utrie
symbolication — it instead shows raw memory addresses (with similarly huge offsets).Unknowns:
Helpful links
Until we put out a new podspec/release with a fix for this, here’s a workaround:
bcsymbolmap
files for the in-use version of Mapbox.framework
— you’ll likely have to download the release again directly from GitHub, as CocoaPods discards these during installation.bcsymbolmap
files as inputs for this phase.You can verify that this worked by looking in the YourApp.xcarchive/BCSymbolMaps
folder.
bcsymbolmap
files when installing the pod, but does not handle the copying of these files into the archived app — for how to do that yourself, see https://github.com/mapbox/mapbox-gl-native/pull/12257#issuecomment-401209457. This is available from ios-v4.2.0-alpha.1
onward.As for properly symbolicating crashes in apps that were submitted without our bcsymbolmap
files, unfortunately I wasn’t able to find a way to do this.
This issue has been automatically detected as stale because it has not had recent activity and will be archived. Thank you for your contributions.
Still CocoaPods fail to load these files even though these are there in Mapbox-iOS-SDK dynamic folder. Any one know fix? by commenting these lines in project-name-framewroks.sh works but that is not a solution install_bcsymbolmap "${PODS_ROOT}/Mapbox-iOS-SDK/dynamic/826E141E-8875-3C3B-A106-2B772F8A0684.bcsymbolmap" install_bcsymbolmap "${PODS_ROOT}/Mapbox-iOS-SDK/dynamic/D2C1B547-E1DE-365D-A6EF-DB16EE1D4375.bcsymbolmap"
@waris117 See https://github.com/mapbox/mapbox-gl-native/pull/12257#issuecomment-401209457 for our suggested workaround. The CocoaPods team is working on fixing this in 1.7.0 (https://github.com/CocoaPods/CocoaPods/pull/8470), which will eventually obviate these extra steps.
This is same issue, it shows missing reports. using xcode 10 Fabric crash reports. while someother crash reports are very accurate. like below
@khaskheliAyaz That appears to be a missing dSYM, rather than a partially-unsymbolicated crash report (as this issue covers) — see Crashlytics’ missing dSYM help page for advice on how to symbolicate that crash.
Platform: iOS Mapbox SDK version: v3.4.2, v3.5.0-beta.3, v3.5.0-rc.1
Some crash reports are only partially symbolicating. Known affected versions of the SDK are listed, but this probably is not limited to those.
If the unsymbolicated crash report retains the unsymbolicated addresses, the dSYM for that version of Mapbox.framework can still be used to symbolicate. But, if the crash report comes from Apple via iTunesConnect/Xcode, the unsymbolicated addresses have been replaced with
_hidden
and the report cannot be symbolicated after the fact.Here’s the same crash, in different versions of the same TestFlight-distributed app, bitcode enabled, using different iOS SDK versions:
Symbolicated (v3.5.0-beta.2)
Click to expand crash log
``` Incident Identifier: F768474A-875F-4EF2-84F7-C5F802773001 Beta Identifier: 7F5ABD3B-D7F7-4809-818E-D0D530573657 Hardware Model: iPhone8,1 Process: Sirius [425] Path: /private/var/containers/Bundle/Application/6547CC84-6BF2-406E-A1B3-F944C858DF4E/Sirius.app/Sirius Identifier: com.mapbox.Sirius Version: 99 (2.0) Beta: YES Code Type: ARM-64 (Native) Role: Foreground Parent Process: launchd [1] Coalition: com.mapbox.Sirius [448] Date/Time: 2017-03-16 23:26:12.0830 -0400 Launch Time: 2017-03-15 08:30:06.9466 -0400 OS Version: iPhone OS 10.2.1 (14D27) Report Version: 104 Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000000 Termination Signal: Segmentation fault: 11 Termination Reason: Namespace SIGNAL, Code 0xb Terminating Process: exc handler [0] Triggered by Thread: 0 Thread 0 name: Thread 0 Crashed: 0 libc++.1.dylib 0x000000018060e37c std::__1::promiseUnsymbolicated (v3.5.0-rc.1)
Click to expand crash log
``` Incident Identifier: 131DD4BD-E90E-42AA-B010-C78D0D6B0432 Beta Identifier: 1FF4C2D1-038C-410E-B618-7A88D74AAE9E Hardware Model: iPhone9,3 Process: Sirius [9952] Path: /private/var/containers/Bundle/Application/638C221C-5DBA-479E-94DB-77DE312F6A0A/Sirius.app/Sirius Identifier: com.mapbox.Sirius Version: 100 (2.0) Beta: YES Code Type: ARM-64 (Native) Role: Foreground Parent Process: launchd [1] Coalition: com.mapbox.Sirius [2663] Date/Time: 2017-03-17 13:36:27.1921 -0700 Launch Time: 2017-03-17 13:15:05.0907 -0700 OS Version: iPhone OS 10.2.1 (14D27) Report Version: 104 Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000000 Termination Signal: Segmentation fault: 11 Termination Reason: Namespace SIGNAL, Code 0xb Terminating Process: exc handler [0] Triggered by Thread: 0 Thread 0 name: Thread 0 Crashed: 0 libc++.1.dylib 0x000000018d1c637c std::__1::promise/cc @mapbox/ios