search

firebase / firebase-unity-sdk

The Firebase SDK for Unity
http://firebase.google.com
Apache License 2.0
201 stars 33 forks source link

[Bug] Crashlytics/Crashes/Events has messed-up stacktraces and no thread name #1055

Closed Zachary625 closed 2 days ago

Zachary625 commented 5 days ago

Description

This is my first time opening an issue to Firebase repo so apologizes for any ill-formated writting in advance, and gratitudes to anyone helping! 😃

Overview:

Crashlytics is showing fatal crashes with stacktraces containing always 64 frames, with many stackframes containing pc values that does not make any sense inserted (not sure if they are from other threads), and the stacktrace records are missing thread names.

Motivation for or Use Case

We're using Crashlytics to monitor our app's vitalities and performances, we've just upgraded our Firebase Unity SDK version and after the upgrade the stacktraces of crashes are nearly uncomprehendable, thus blocking our work.

Operating System

Observed on both iOS and Android, after several rounds of inspection on Android I managed to reproduce the issue using a demo Unity + Firebase Crashlytics project, then lead to the hypothesis of Crashlytics issue.

Observations:

An example case crash reproduced by method below resulted in a tombstone like this:

Version '2022.3.18f1 (d29bea25151d)', Build type 'Release', Scripting Backend 'il2cpp', CPU 'arm64-v8a'
Build fingerprint: 'HUAWEI/BLA-AL00/HWBLA:10/HUAWEIBLA-AL00/10.0.0.175C00:user/release-keys'
Revision: '0'
ABI: 'arm64'
Timestamp: 2024-07-02 17:24:43.562668187+0800
pid: 32598, tid: 32665, name: UnityMain  >>> TEST_APP_ID <<<
uid: 11321
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr --------
Cause: null pointer dereference
    x0  0000000000000001  x1  0000000000000000  x2  0000000000000000  x3  0000006f31b4ea37
    x4  0000000000000206  x5  4008000000000000  x6  0000000000000000  x7  7f7f7f7f7f7f7f7f
    x8  0000000000000001  x9  01b5a63d3e722294  x10 0000000000000000  x11 0000000000000000
    x12 0000000000000000  x13 ffffffffffffffff  x14 0000000000000004  x15 ffffffffffffffff
    x16 0000006f324f8e90  x17 000000703991ef80  x18 0000006ef1106bc8  x19 0000000000000001
    x20 0000006f317ca210  x21 0000006f97c8cb28  x22 0000006e540bb930  x23 0000006f317ced60
    x24 0000006f317ca050  x25 0000006f317c88f0  x26 0000006f317c6de8  x27 0000000000000001
    x28 0000006f317deb70  x29 0000006f97ca1dd0
    lr  0000006f3134af20  sp  0000006f3efe7ec0  pc  0000006f3134af20  pst 0000000060000000

backtrace:
      #00 pc 0000000000ffbf20  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #01 pc 00000000009c2370  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #02 pc 000000000119bfac  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #03 pc 0000000000aea540  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #04 pc 0000000001385310  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #05 pc 0000000001384908  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #06 pc 0000000001384668  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #07 pc 00000000008cf0dc  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #08 pc 00000000008cf028  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #09 pc 0000000000419be4  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
      #10 pc 000000000042929c  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
      #11 pc 0000000000436640  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
      #12 pc 00000000002d65f8  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
      #13 pc 00000000003676d0  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
      #14 pc 0000000000367710  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
      #15 pc 00000000003679a4  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
      #16 pc 00000000004a872c  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
      #17 pc 00000000004bd9a8  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
      #18 pc 000000000002a26c  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/oat/arm64/base.odex

while Crashlytics recorded and reported this:

Crashed: Thread: SIGSEGV  0x0000000000000001
#00 pc 0xffbf20 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#01 pc 0x9c2370 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#02 pc 0x148b044 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#03 pc 0x119bfac libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#04 pc 0xaea540 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#05 pc 0x1480304 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#06 pc 0x26a7ac libunity.so (BuildId: f78b65a4b65058ee)
#07 pc 0x1385310 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#08 pc 0x147b584 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#09 pc 0x147a67c libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#10 pc 0x1507ffc libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#11 pc 0x1384908 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#12 pc 0xaeb2bc libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#13 pc 0x1384668 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#14 pc 0x8cf0dc libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#15 pc 0x8cf028 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#16 pc 0x419be4 libunity.so (BuildId: f78b65a4b65058ee)
#17 pc 0xbbd34 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#18 pc 0x42929c libunity.so (BuildId: f78b65a4b65058ee)
#19 pc 0xaf6814 libunity.so (BuildId: f78b65a4b65058ee)
#20 pc 0x436640 libunity.so (BuildId: f78b65a4b65058ee)
#21 pc 0xcb231 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#22 pc 0x3d6958 libunity.so (BuildId: f78b65a4b65058ee)
#23 pc 0x2701b0 libunity.so (BuildId: f78b65a4b65058ee)
#24 pc 0x3d6958 libunity.so (BuildId: f78b65a4b65058ee)
#25 pc 0x2d65f8 libunity.so (BuildId: f78b65a4b65058ee)
#26 pc 0x3676d0 libunity.so (BuildId: f78b65a4b65058ee)
#27 pc 0x4da860 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#28 pc 0x119ea60 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#29 pc 0x1505ffc libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#30 pc 0x8cf0dc libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#31 pc 0x8cf028 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#32 pc 0x419be4 libunity.so (BuildId: f78b65a4b65058ee)
#33 pc 0x4d2fd0 libunity.so (BuildId: f78b65a4b65058ee)
#34 pc 0x367710 libunity.so (BuildId: f78b65a4b65058ee)
#35 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#36 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#37 pc 0x42929c libunity.so (BuildId: f78b65a4b65058ee)
#38 pc 0x6f4a7dc72e
#39 pc 0xaf6814 libunity.so (BuildId: f78b65a4b65058ee)
#40 pc 0x9c115 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#41 pc 0x3679a4 libunity.so (BuildId: f78b65a4b65058ee)
#42 pc 0x6f4a7dc72e
#43 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#44 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#45 pc 0xaebffc libunity.so (BuildId: f78b65a4b65058ee)
#46 pc 0x6f4a7dc72e
#47 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#48 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#49 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#50 pc 0x49aef8 libunity.so (BuildId: f78b65a4b65058ee)
#51 pc 0x4a872c libunity.so (BuildId: f78b65a4b65058ee)
#52 pc 0x4a8704 libunity.so (BuildId: f78b65a4b65058ee)
#53 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#54 pc 0xa76ffc libunity.so (BuildId: f78b65a4b65058ee)
#55 pc 0x4bd9a8 libunity.so (BuildId: f78b65a4b65058ee)
#56 pc 0x6f4a18626c
#57 pc 0x4bd950 libunity.so (BuildId: f78b65a4b65058ee)
#58 pc 0x6f4a7dc72e
#59 pc 0x148334 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#60 pc 0x6f4a7dc72e
#61 pc 0x1571b4 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#62 pc 0x2fe588 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#63 pc 0x60affc libart.so (BuildId: 73a2145672853571ef40097e4441d44e)

After merging the tombstone stacktrace into the crashlytics stacktrace, it resulted like this(tabbed lines are from tombstone)

Crashed: Thread: SIGSEGV  0x0000000000000001
#00 pc 0xffbf20 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #00 pc 0000000000ffbf20  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#01 pc 0x9c2370 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #01 pc 00000000009c2370  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#02 pc 0x148b044 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#03 pc 0x119bfac libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #02 pc 000000000119bfac  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#04 pc 0xaea540 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #03 pc 0000000000aea540  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#05 pc 0x1480304 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#06 pc 0x26a7ac libunity.so (BuildId: f78b65a4b65058ee)
#07 pc 0x1385310 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #04 pc 0000000001385310  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#08 pc 0x147b584 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#09 pc 0x147a67c libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#10 pc 0x1507ffc libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#11 pc 0x1384908 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #05 pc 0000000001384908  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#12 pc 0xaeb2bc libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#13 pc 0x1384668 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #06 pc 0000000001384668  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#14 pc 0x8cf0dc libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #07 pc 00000000008cf0dc  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#15 pc 0x8cf028 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
      #08 pc 00000000008cf028  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#16 pc 0x419be4 libunity.so (BuildId: f78b65a4b65058ee)
      #09 pc 0000000000419be4  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
#17 pc 0xbbd34 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#18 pc 0x42929c libunity.so (BuildId: f78b65a4b65058ee)
    #10 pc 000000000042929c  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
#19 pc 0xaf6814 libunity.so (BuildId: f78b65a4b65058ee)
#20 pc 0x436640 libunity.so (BuildId: f78b65a4b65058ee)
      #11 pc 0000000000436640  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
#21 pc 0xcb231 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#22 pc 0x3d6958 libunity.so (BuildId: f78b65a4b65058ee)
#23 pc 0x2701b0 libunity.so (BuildId: f78b65a4b65058ee)
#24 pc 0x3d6958 libunity.so (BuildId: f78b65a4b65058ee)
#25 pc 0x2d65f8 libunity.so (BuildId: f78b65a4b65058ee)
      #12 pc 00000000002d65f8  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
#26 pc 0x3676d0 libunity.so (BuildId: f78b65a4b65058ee)
      #13 pc 00000000003676d0  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
#27 pc 0x4da860 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#28 pc 0x119ea60 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#29 pc 0x1505ffc libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#30 pc 0x8cf0dc libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#31 pc 0x8cf028 libil2cpp.so (BuildId: b56bb355577dd0fc45d2c77a0ca8a3e46034e059)
#32 pc 0x419be4 libunity.so (BuildId: f78b65a4b65058ee)
#33 pc 0x4d2fd0 libunity.so (BuildId: f78b65a4b65058ee)
#34 pc 0x367710 libunity.so (BuildId: f78b65a4b65058ee)
      #14 pc 0000000000367710  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
#35 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#36 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#37 pc 0x42929c libunity.so (BuildId: f78b65a4b65058ee)
#38 pc 0x6f4a7dc72e
#39 pc 0xaf6814 libunity.so (BuildId: f78b65a4b65058ee)
#40 pc 0x9c115 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#41 pc 0x3679a4 libunity.so (BuildId: f78b65a4b65058ee)
      #15 pc 00000000003679a4  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
#42 pc 0x6f4a7dc72e
#43 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#44 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#45 pc 0xaebffc libunity.so (BuildId: f78b65a4b65058ee)
#46 pc 0x6f4a7dc72e
#47 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#48 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#49 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#50 pc 0x49aef8 libunity.so (BuildId: f78b65a4b65058ee)
#51 pc 0x4a872c libunity.so (BuildId: f78b65a4b65058ee)
      #16 pc 00000000004a872c  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
#52 pc 0x4a8704 libunity.so (BuildId: f78b65a4b65058ee)
#53 pc 0xaf9ffc libunity.so (BuildId: f78b65a4b65058ee)
#54 pc 0xa76ffc libunity.so (BuildId: f78b65a4b65058ee)
#55 pc 0x4bd9a8 libunity.so (BuildId: f78b65a4b65058ee)
      #17 pc 00000000004bd9a8  /data/app/TEST_APP_ID-CALRGQvxJHtBviUjhU_iEA==/lib/arm64/libunity.so (BuildId: f78b65a4b65058ee)
#56 pc 0x6f4a18626c
#57 pc 0x4bd950 libunity.so (BuildId: f78b65a4b65058ee)
#58 pc 0x6f4a7dc72e
#59 pc 0x148334 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#60 pc 0x6f4a7dc72e
#61 pc 0x1571b4 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#62 pc 0x2fe588 libart.so (BuildId: 73a2145672853571ef40097e4441d44e)
#63 pc 0x60affc libart.so (BuildId: 73a2145672853571ef40097e4441d44e)

Reproducing the issue

Minimal Reproducible Example can be provided on demand

Environment

Unity 2022.3.18f1 with Firebase Crashlytics Unity SDK 11.8.1 using default Android BoM 32.7.4, build APK directly with IL2CPP as scripting backend using Gradle build system.

This is found after we rolled out an upgrade, from Firebase Unity SDK 8.0.0 to 11.8.1(reverse-engineered to adapt .NET 3.5 runtime), our production project is using Unity 2018.,4.11f1 with .NET 3.5.

Testing with downgraded Firebase SDKs are in comment section

Implementation

Initializing Firebase

        FirebaseApp.LogLevel = LogLevel.Verbose;
        FirebaseApp.CheckAndFixDependenciesAsync().ContinueWith(task =>
        {
            DependencyStatus status = task.Result;
            Debug.Log("CheckAndFixDepsAsync => " + status.ToString());

            if(status == DependencyStatus.Available)
            {
                FirebaseApp firebaseApp = FirebaseApp.DefaultInstance;
                Crashlytics.IsCrashlyticsCollectionEnabled = true;
                Crashlytics.SetUserId("TestUser");
                Debug.Log("Firebase Initialization Complete, proceed to testing please.");
            }
        });

Triggering a crash

    public void OnCrashClick()
    {
        Debug.Log("+ OnCrashClick");
        Marshal.WriteByte(new IntPtr(1), 1);
        Debug.Log("- OnCrashClick");
    }

Firebase Unity SDK Version

11.8.1

Unity editor version

2022.3.18f1

Installation Method

.unitypackage

Problematic Firebase Component(s)

Crashlytics

Other Firebase Component(s) in use

No response

Additional SDKs you are using

No response

Targeted Platform(s)

Apple Platforms, Android

Unity editor platform

Windows

Scripting Runtime

IL2CPP

Release Distribution Type

Pre-built SDK from https://firebase.google.com/download/unity

Relevant Log Output

No response

If using CocoaPods for Apple platforms, the project's Podfile.lock

Expand Podfile.lock snippet
```yml 👀 Replace this line with the contents of your Podfile.lock! ```
Zachary625 commented 5 days ago

Update of futher testing, all with IL2CPP and ARM32 & ARM64 enabled

Looking forward for replies, thanks.

argzdev commented 3 days ago

Hey @Zachary625, thanks for this well detailed report. I've tested with Unity 2022.3.24f1 + Firebase Unity SDK 11.8.1 with the Firebase Unity Quickstart, however I'm not encountering the same issue. Just as a sanity check, could you confirm if you've uploaded the symbols after building the app?

Zachary625 commented 3 days ago

Hi thank you @argzdev, interestingly, uploading the symbols (with Firebase CLI windows-x64) would change the stacktrace to two short lines, sth like this:

          Crashed: Thread: SIGSEGV  0x0000000000000001
#00 pc 0xffc724 libil2cpp.so (Marshal.WriteByte [mscorlib__12.cpp]) (BuildId: 5a31f8a7a0115b05bba2704988f06d92e0e8937e)
#01 pc 0x26a7ac libunity.so (Component_Get_Custom_PropGameObject(ScriptingBackendNativeObjectPtrOpaque*)) (BuildId: f78b65a4b65058ee)

#00 being the correct #00 in tomestone and stacktrace, while #01 pc value is never present in the tombstone, seems like uploading the symbol changed the way crashlytics(guessing server) interprets the stacktrace, this is an improvement but I believe it should show up more frames between the two lines, at least my test script should be there and highlighted.

I'm working on 2 directions:

  1. using Unity 2018.4.11f1 and legacy Firebase 8.0.0 (ndk) as a rollback option, which we were using previously, but seems like neither our project nor the test project is showing native crashes (confirmed managed exceptions can show up almost instantly).
    • Uneducated guess is that it is because the crashlytics backend detects we're already using 11.8.1, which supports crash-free user sessions that crashlytics promoted as a nice feature and perhaps requires a brand new backend system replacing the old one, thus automatically ignored native crashes from 8.0.0.
  2. still trying to figure out Unity 2022.3.18f1 + Firebase 11.8.1 NDK version IL2CPP reporting nonsense stacktraces, I see there are tomestone files dumped, but a bit searching in firebase sdk source code told me firebase is caching tombstones in fire store? not sure how we can easily inspect the internal details of caching and uploading. I'm using firebase log level verbose and adb shell setprop log.tag.CrashlyticsCore DEBUG all i see after bootstrap self-inspection logs is: 
    2024-07-04 22:51:15.711 28559-28612 TRuntime.Cc...ortBackend TEST_APP_ID      I  Making request to: https://crashlyticsreports-pa.googleapis.com/v1/firelog/legacy/batchlog
// bootstrap self-inspection stuff
2024-07-04 22:51:19.308 28559-28612 TRuntime.Cc...ortBackend TEST_APP_ID      I  Status Code: 200
2024-07-04 22:51:45.741 28559-28612 TRuntime.Cc...ortBackend TEST_APP_ID      I  Making request to: https://firebaselogging-pa.googleapis.com/v1/firelog/legacy/batchlog
2024-07-04 22:51:47.161 28559-28612 TRuntime.Cc...ortBackend TEST_APP_ID      I  Status Code: 200

More details of the build:

Once again thank you for working on this matter!

argzdev commented 3 days ago

Good to hear that there's an improvement. There's quite a bit of detail here. I'll try to address some of the provided information:

I believe you should see more frames than just 2 short lines. Here's an example of the stacktrace on my side based on the initial code you've provided:

          Crashed: Thread: SIGSEGV  0x0000000000000001
#00 pc 0x7688b9 libil2cpp.so (Marshal.WriteByte [mscorlib__12.cpp]) (BuildId: 065b0b24cb272764c6b528c459294ce81983b740)
#01 pc 0x5a6f3f libunity.so (GUIClipState::Apply(InputEvent&)) (BuildId: be2fca746b83e65c)
#02 pc 0xb13720 libunity.so (__unw_getcontext) (BuildId: be2fca746b83e65c)
#03 pc 0x43d443 libunity.so (scripting_method_invoke(ScriptingMethodPtr, ScriptingObjectPtr, ScriptingArguments&, ScriptingExceptionPtr*, bool)) (BuildId: be2fca746b83e65c)
#04 pc 0xb13558 libunity.so (__unw_getcontext) (BuildId: be2fca746b83e65c)
#05 pc 0x44aa3e libunity.so (ScriptingInvocation::Invoke(ScriptingExceptionPtr*, bool)) (BuildId: be2fca746b83e65c)
#06 pc 0x5a54d7 libunity.so (MonoBehaviourDoGUI(int, ObjectGUIState&, MonoBehaviour::GUILayoutType, int, ScriptingMethodPtr, PPtr<MonoBehaviour>)) (BuildId: be2fca746b83e65c)
#07 pc 0xb309f0 libunity.so (__unw_getcontext) (BuildId: be2fca746b83e65c)
#08 pc 0x5a6f3f libunity.so (GUIClipState::Apply(InputEvent&)) (BuildId: be2fca746b83e65c)
#09 pc 0xa79ec0 libunity.so (__unw_getcontext) (BuildId: be2fca746b83e65c)
#10 pc 0x457614 libunity.so (MonoBehaviour::DoGUI(MonoBehaviour::GUILayoutType, int, int)) (BuildId: be2fca746b83e65c)
....
....
#60 pc 0xca056 libc.so (BuildId: fa337969c798946280caa45e2d71a2e7)
#61 pc 0xcd06b libc.so (BuildId: fa337969c798946280caa45e2d71a2e7)
#62 pc 0xcd030 libc.so (BuildId: fa337969c798946280caa45e2d71a2e7)
#63 pc 0x62d89 libc.so (BuildId: fa337969c798946280caa45e2d71a2e7)

Could you check your Build Settings, if "Debugging" option is selected for the dropdown in the Create symbol.zip. According to the docs, if "Public" option is selected, this will create public symbols that remove more in-depth debug information.

Uneducated guess is that it is because the crashlytics backend detects we're already using 11.8.1, which supports crash-free user sessions that crashlytics promoted as a nice feature and perhaps requires a brand new backend system replacing the old one, thus automatically ignored native crashes from 8.0.0.

IIRC, after installing the updated SDK and uploading symbol information, subsequently-received native crashes should be symbolicated correctly. Previously-received crashes will remain unsymbolicated, as will crashes from older versions of the app that have earlier Firebase Unity versions.

a bit searching in firebase sdk source code told me firebase is caching tombstones in fire store

Crashlytics have its dedicated storage mechanism for logs and crash reports. I'm not aware of it storing in Firestore, any chance you could share where in the source code did you see this?

While I understand there's a lot of gray area that we haven't covered. This article from Unity support might help shed light on certain points.

Let me know if these help.

Zachary625 commented 2 days ago

Thanks @argzdev, this is a quick response, will edit if more details/changes emerge before you reply:

  1. this stacktrace you managed to reproduce indeed looks like what we'd expect in the beginning 10 frames, but we can only symbolicate the beginning part locally, which are the full stacktrace we get in LogCat and tombstone, when crashlytics got uploaded symbols it shows only two lines. Couple of observations on your stacktrace:
    • Why are you seeing 64 frames too? IIRC in several years of seeing crashes on Crashlytics every now and then, a common native crash is often [10, 50] frames deep. We're only seeing 64 frames of stacktraces only after we upgraded from 8.0.0 to 11.8.1, if this is expected then we'd be ignoring this depth anomally.
    • I guess you're using IMGUI to call the Marshal.WriteByte, whic I'll be trying too, but the (__unw_getcontext) is new to me, and perhaps if you could use UGUI to bind a method to a button to reproduce, just for more resemblence? Our bottom line is to get accurate stacktraces, raw is tolerable.
      • Update: tried Unity2022 + 11.8.1NDK with IMGUI + Marshal.WriteByte(1, 1), result symbolicated being:
Crashed: Thread: SIGSEGV  0x0000000000000001
#00 pc 0x742fcc libil2cpp.so (__emutls_get_address) (BuildId: f008cccf48b403dbfc3dc556c06e12e3ac32c4cd)
#01 pc 0x742fc8 libil2cpp.so (__emutls_get_address) (BuildId: f008cccf48b403dbfc3dc556c06e12e3ac32c4cd)

still faulty I reckon.

TSET_APP_ID_issue_84ce77d64742ac0cae6fcd37629407de_crash_session_668798b700c300013ddc5eb51d19f162_DNE_0_v2_stacktrace.txt

  1. Create symbols.zip is selected to debugging I confirm, exported gradle project only has unityLibrary/symbols/arm64-v8a & armeabi-v7a/libil2cpp.so & libmain.so & libunity.so and no zip, I pass this symbols folder path directly to Firebase CLI to upload, yielding success outputs.
    • Update: Per guideline from Crashlytics/NDK-Reports I also tried adding android.buildTypes.release.firebaseCrashlytics.nativeSymbolUploadEnabled true and use gradlew launcher:assembleRelease launcher:uploadCrashlyticsSymbolFileRelease, no luck, and use readelf to check libunity.so and libil2cpp.so stripped and symbol elf files and confirm the GNU build id in readelf output, logcat output, and crashlytics console are all the same(glad to see one thing correct at least).
  2. Thanks for explaining on symbolcating mechanism, but 8.0.0's reports of native crashes never show up for either project is what bothers me, as this blocks our retreat path, are there mechanisms in crashlytics that will drop/ignore old version native crashes if new versions are detected in use, between 8.0.0 and 11.8.1?
  3. Sorry I was wrong it was https://github.com/firebase/firebase-android-sdk/blob/main/firebase-crashlytics/src/main/java/com/google/firebase/crashlytics/internal/persistence/FileStore.java#L52 so ignore me.

I'm reinstalling Unity 2022 and redownloading Firebase 11.8.1 to get a clean setup, and change version name & code to perhaps force crashlytics to change the BuildId.

Update:

Inspired by Crashlytics Troubleshooting I changed C++ Compiler Configuration from Release to Debug in Unity 2022 and Crashlytics native crash stacktrace and symbolication is working (-Wl,--no-rosegment has no effect). Investigating more.

argzdev commented 2 days ago

Thanks for the confirming and adding additional details, @Zachary625.

void GUIDisplayControls() { ... if (GUILayout.Button("OnCrashClick")) { OnCrashClick(); } }



> - libc.so, libbinder.so, libutils.so, libandroid_runtime.so, libart.so, libandroid.so all missing build id

I believe this is expected. Stacktraces that are associated with "libc.so" are Android system runtime libraries. Crashlytics is unable to symbolicate those libraries since we don't have visibility into the symbols associated with the system libraries. We're working on making this better in the future but there's no timeline for this.

2. Thanks for confirming this. There's a slight discrepancy in the path for uploading the symbol, according to the [docs](https://firebase.google.com/docs/crashlytics/get-started?platform=unity#build-project-upload-symbols-android), using the path `unityLibrary/symbols` should be enough. However, I don't think this difference should cause any issue as long as all symbols are uploaded.

> are there mechanisms in crashlytics that will drop/ignore old version native crashes if new versions are detected in use, between 8.0.0 and 11.8.1?

3. Crashlytics shouldn't ignore or drop old crashes. It stores the crash reports in the backend infrastructure, so upgrading the SDK version should not affect the data storage. Plus, Crashlytics was made to be backward compatible, so it should still be able to access previously encountered crash reports. Sounds like there could be a problem with the backend.

In summary, I don't think there's any issue with your configuration on the SDK. I believe the issue you're encountering might have something to do with the backend. In this case, could you reach out to our [Firebase support](https://firebase.google.com/support), since we from the SDK team do not have access to your project details. The support team will be able to take a look into your project configuration. Please do share with them this thread, logs, your minimal reproducible example, and any other information you think might be helpful for them. 

That said, I'll go ahead and close this thread. Feel free to come back here if you assume that the issue might still be due to the SDK, and we'll reopen this thread right away. Thanks!
Zachary625 commented 2 days ago

Thank you @argzdev ! 1.PlayerSettings.SetAdditionalIl2CppArgs("-compiler-flags=\"-fno-omit-frame-pointer\""); would do the trick. Also curiously Firebase 8.0.0 had no such issue, only 11.8.1 came with this issue, so it might be changes in SDK side?

  1. Also in Unity 2018 setting release to debug also works, to make symbolication work for libil2cpp we have to change name of generated libil2cpp.sym to libil2cpp.sym.so
  2. Still, why is 8.0.0 not uploading any native crash might be of some importance
argzdev commented 2 days ago

Hi @Zachary625, you could be right regarding the SDK issue. So the reason there was a change in behavior from version 8.0.0 is because starting with Firebase Unity 8.6.1, Crashlytics will catch and report native IL2CPP crashes for Unity apps. To properly receive native IL2CPP crashes for Unity apps uploading of native library symbol information that is generated at build time is now required.

Now, we might need an in depth analysis of the issue. That said, since we don't have access to project details. Could you file a report to our Firebase support team with the following details:

Also please refer them to this discussion thread. Our engineering team will be able to take a look into your project configs.

Zachary625 commented 2 days ago

Thank you @argzdev, so just to confirm I understand correctly, seeing as how we could get native crashes when we were using 8.0.0 and now that we started on 11.8.1 then old 8.0.0 builds cannot receive any native crashes, what you're saying is that if crashlytics (backend?) detects we be using sdk newer than 8.6.1, it would hide/drop natve crashes sent by older SDKs if no corresponding symbol files that was uploaded? This is indeed a possible explanation and I'll be working on validating this. And perhaps there'd be a release note or doc or sth to backup this hypothesis?

argzdev commented 2 days ago

Hi @Zachary625, there might've been some misunderstanding. What I mean is there are some changes with the versions newer than 8.6.1 which leads to missing frames or unsymbolicated stacktrace(which can be fixed by uploading the symbols via CLI). However, this does not explain why older builds cannot receive native crashes. Crashlytics should still receive and show any native crashes of any version including older builds, since it should not have an automatic ignore feature. Which is why I encourage you to submit a report to our Firebase support team, so our engineering team can dig deeper regarding the issue.

Zachary625 commented 1 day ago

Thanks for the clarification! Much obliged!