Crash stack trace cannot be symbolicated, because the debug images do not contain necessary pieces

[fzyzcjy]

Description

Hi thanks for the library! However, crash stack trace cannot be symbolicated, because the debug images do not contain necessary pieces.

I checked the JSON, and it seems that it is missing some important pieces. More specifically, the .debugMeta.images section only have system things (e.g. /vendor/lib/libmemalloc.so) and some of the app so (e.g. /data/app/com.cjy.yplusplus-1/lib/arm/libsentry.so). But it does not have my libvision_utils.so (a Rust dynamic library I wrote for my app), neither does it have libapp.so or libflutter.so (which definitely should exist and be used in my Flutter app!).

I hope I am getting wrong and wish I can symbolicate these frames!

The JSON is:

      {
        "type": "elf",
        "debug_id": "0136f226-2b0f-0917-4a86-0598772e82f6",
        "code_id": "26f236010f2b17094a860598772e82f6d3f6d6c1",
        "code_file": "/data/app/com.cjy.yplusplus-1/lib/arm/libsentry.so",
        "image_addr": "0xa7bd1000",
        "image_size": 712704
      },

...

            {
              "absPath": null,
              "addrMode": null,
              "colno": null,
              "contextLine": null,
              "filename": null,
              "function": null,
              "functionId": null,
              "imageAddr": "0x8f964000",
              "inApp": null,
              "instructionAddr": "0x8fd18de3",
              "lineno": null,
              "lock": null,
              "module": null,
              "package": "/data/app/com.cjy.yplusplus-1/lib/arm/libvision_utils_rs.so",
              "platform": null,
              "postContext": null,
              "preContext": null,
              "rawFunction": null,
              "stackStart": null,
              "symbol": null,
              "symbolAddr": null,
              "vars": null
            },

...

  "sdk": {
    "integrations": [
      "inproc"
    ],
    "name": "sentry.native.android",
    "packages": [
      {
        "name": "github:getsentry/sentry-native",
        "version": "0.6.3"
      }
    ],
    "version": "0.6.3"
  },
...

When does the problem happen

• [ ] During build
• [ ] During run-time
• [ ] When capturing a hard crash

Environment

• OS: [e.g. Windows 10, 64-bit]
• Compiler: [e.g. MSVC 19]
• CMake version and config: [e.g. 3.17.2, SENTRY_BACKEND=inproc]

Steps To Reproduce

Log output

[supervacuus]

Hi @fzyzcjy! Did you upload the debug-info of all your binaries according to the docs?

[fzyzcjy]

@supervacuus Hi, IMHO it is missing critical information, please correct me if I am wrong!

Usually, the imageAddr is used to determine which .so's symbol should be used. However, here the imageAddr of the stack frame does not correspond to any debug image, so it cannot be decoded.

[supervacuus]

@fzyzcjy, thanks. I misunderstood the issue you were raising. You're saying that the sentry backend retrieves an incomplete list of modules from the Android device? Is this reproducible, or do you have this only from a singular event? In any case, it would help if you could provide an event URL for someone from the team to inspect the whole event.

Typically the image list is incomplete when you are using an obfuscator that also touches the ELF files and makes them unusable for our module-finder or obfuscates the debug-id (https://github.com/getsentry/sentry-native/issues/849). There were also issues when users sent a non-crash event before a later crash, where in-between modules were loaded, since we cache the module-list after an initial load.

But it is hard to get an idea of what is happening without seeing the full event.

[fzyzcjy]

You're saying that the sentry backend retrieves an incomplete list of modules from the Android device?

I guess yes.

Is this reproducible, or do you have this only from a singular event?

Not tried to reproduce yet, but I will report if it reproduces.

In any case, it would help if you could provide an event URL for someone from the team to inspect the whole event.

Unfortunately I cannot use the Sentry cloud service even though I want to use it :( (See https://github.com/getsentry/sentry/issues/51859 for why) But I can send the event to the team! Just give me an email address (or other communication methods).

Typically the image list is incomplete when you are using an obfuscator that also touches the ELF files and makes them unusable for our module-finder or obfuscates the debug-id (https://github.com/getsentry/sentry-native/issues/849). There were also issues when users sent a non-crash event before a later crash, where in-between modules were loaded, since we cache the module-list after an initial load.

Thank you for the information!

[kahest]

Hey @fzyzcjy thanks for following up - you can send the event to karl.struggl@sentry.io

[fzyzcjy]

Hi, thanks for the reply and sorry for the delay, I have sent the event JSON data to your email

[kahest]

@fzyzcjy no worries, thanks for following up, we will take a look

[fzyzcjy]

You are welcome! Looking forward to it

[supervacuus]

Hi @fzyzcjy! I might not have been clear enough with this statement:

There were also issues when users sent a non-crash event before a later crash, where in-between modules were loaded, since we cache the module-list after an initial load.

Is it possible that your libvision_utils_rs.so (and maybe also libapp.so and libflutter.so) is loaded dynamically later in the execution of your app? In that case, your shared libraries don't appear because we read the module list at one point and cache it, and then later, when you crash, only use that cached version. You can ensure that a refreshed module list is used after loading shared libraries by calling sentry_clear_modulecache().

[fzyzcjy]

Is it possible that your libvision_utils_rs.so (and maybe also libapp.so and libflutter.so) is loaded dynamically later in the execution of your app?

I am using the Flutter framework, and IMHO the procedure is like this:

• android app starts
• libflutter.so is loaded at onCreate (or something like that - very early in the lifetime)
• libapp.so is then immediately loaded
• libvision_utils_rs.so is then loaded, via dynamic loading mechanism

If I understand correctly, the native part is auto inited from the Sentry Android, which seems to be called before flutter system has loaded, so your suggestion may be the cause!

I will try that, thank you!

[getsantry[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀