logcat.log is not attached to unhandled event (SIGSEGV Segfault)

espenrl commented 4 months ago

Package

Sentry.Maui

.NET Flavor

.NET

.NET Version

8.0.6

OS

Android

SDK Version

4.7.0

Self-Hosted Sentry Version

No response

Steps to Reproduce

options.Android.LogCatIntegration = LogCatIntegrationType.Unhandled;

I don't know how to provoke SIGSEGV Segfault.

Expected Result

It is expected that the error event has an attachment logcat.log.

Actual Result

There are no attachments to the event.

espenrl commented 4 months ago

This is one of many of these cases logged to Sentry Cloud. I've redacted it, but left the important bits.

{
  "exception": {
    "values": [
      {
        "mechanism": {
          "handled": false,
          "meta": {
            "signal": {
              "name": "SIGSEGV",
              "number": 11
            }
          },
          "synthetic": true,
          "type": "signalhandler"
        },
        "raw_stacktrace": {
          "frames": [
            {
              "data": {
                "orig_in_app": -1
              },
              "in_app": false,
              "instruction_addr": "0x7270b58418"
            }
          ],
          "registers": {
            "fp": "0x7213c09750",
            "lr": "0x7324b4c84c",
            "pc": "0x7270b58418",
            "sp": "0x7213c09750",
            "x0": "0x7242390a90",
            "x1": "0x72423908b0",
            "x10": "0x1",
            "x11": "0x1",
            "x12": "0xd61f0200",
            "x13": "0xb",
            "x14": "0xb4000074ca2cbf88",
            "x15": "0xb4000074cc6bb5d8",
            "x16": "0x7270b58418",
            "x17": "0x0",
            "x18": "0x721351c000",
            "x19": "0x0",
            "x2": "0x12",
            "x20": "0x72bc7f6bb0",
            "x21": "0x0",
            "x22": "0x0",
            "x23": "0x72bc437d78",
            "x24": "0x7242390a90",
            "x25": "0x72bc491108",
            "x26": "0x72bc437550",
            "x27": "0x7213c0a350",
            "x28": "0x7213c09c50",
            "x3": "0x10",
            "x4": "0x0",
            "x5": "0xb40000735a5785e0",
            "x6": "0xb4000074ca3ffc50",
            "x7": "0x110000",
            "x8": "0x1",
            "x9": "0x0"
          }
        },
        "stacktrace": {
          "frames": [
            {
              "data": {
                "orig_in_app": -1,
                "symbolicator_status": "unknown_image"
              },
              "in_app": false,
              "instruction_addr": "0x7270b58418"
            }
          ],
          "instruction_addr_adjustment": "none",
          "registers": {
            "fp": "0x7213c09750",
            "lr": "0x7324b4c84c",
            "pc": "0x7270b58418",
            "sp": "0x7213c09750",
            "x0": "0x7242390a90",
            "x1": "0x72423908b0",
            "x10": "0x1",
            "x11": "0x1",
            "x12": "0xd61f0200",
            "x13": "0xb",
            "x14": "0xb4000074ca2cbf88",
            "x15": "0xb4000074cc6bb5d8",
            "x16": "0x7270b58418",
            "x17": "0x0",
            "x18": "0x721351c000",
            "x19": "0x0",
            "x2": "0x12",
            "x20": "0x72bc7f6bb0",
            "x21": "0x0",
            "x22": "0x0",
            "x23": "0x72bc437d78",
            "x24": "0x7242390a90",
            "x25": "0x72bc491108",
            "x26": "0x72bc437550",
            "x27": "0x7213c0a350",
            "x28": "0x7213c09c50",
            "x3": "0x10",
            "x4": "0x0",
            "x5": "0xb40000735a5785e0",
            "x6": "0xb4000074ca3ffc50",
            "x7": "0x110000",
            "x8": "0x1",
            "x9": "0x0"
          }
        },
        "type": "SIGSEGV",
        "value": "Segfault"
      }
    ]
  },
  "level": "fatal",
  "location": null,
  "logger": "",
  "message": "",
  "metadata": {
    "display_title_with_tree_label": true,
    "in_app_frame_mix": "system-only",
    "value": "Segfault"
  },
  "platform": "native",
  "sdk": {
    "integrations": [
      "inproc"
    ],
    "name": "sentry.native.dotnet",
    "packages": [
      {
        "name": "github:getsentry/sentry-native",
        "version": "0.7.0"
      }
    ],
    "version": "0.7.0"
  },
  "tags": [
    [
      "handled",
      "no"
    ],
    [
      "installerStore",
      "com.android.vending"
    ],
    [
      "isSideLoaded",
      "false"
    ],
    [
      "level",
      "fatal"
    ],
    [
      "mechanism",
      "signalhandler"
    ],
    [
      "os",
      "Android 12"
    ],
    [
      "os.name",
      "Android"
    ],
    [
      "os.rooted",
      "no"
    ]
  ],
  "title": "<unknown>",
  "type": "error",
  "version": "7"
}

espenrl commented 4 months ago

It's important to note that some unhandled events do get logcat.log attached, so the mechanism do work. There seems to be a difference between exceptions originating from .NET and those originating from native code. The latter ones doesn't get logcat.log attached. And it's not just one event. We get lots of these.

bitsandfoxes commented 4 months ago

Interesting! thanks @espenrl for reaching out! So the logcat.log only get added as attachment within the managed code. That should be easy enough to reproduce!

espenrl commented 4 months ago

@bitsandfoxes It may simply be the fact that when Mono enters mono_handle_native_crash it prints a (rather good) crash report to stdout and then exits the program. Perhaps Sentry doesn't get a chance to process anything, and the event is actually sent at the next startup.

https://github.com/dotnet/runtime/blob/main/src/mono/mono/mini/mini-exceptions.c#L2921

On a side note: stacktraces in those native crashes aren't easily symbolicated. Only Mono has sufficient information how to unwind it. Meaning it is best to capture the crash report printed by Mono to stdout. On Android stdout is pointing to /dev/null and it is possible to redirect it and write it to file. Please see this issue https://github.com/dotnet/android/issues/8832

bitsandfoxes commented 4 months ago

Thanks for the additional context!

jamescrosswell commented 3 months ago

This might be related to:

https://github.com/dotnet/android/issues/9055#issuecomment-2261347912

espenrl commented 3 months ago

This might be related to:

SIGSEGV reported by Sentry when a NullReferenceException is thrown in Release mode dotnet/android#9055 (comment)

Very interesting read. That would mean that those issues that gets logged to Sentry aren't necessarily native crashes at all. Good to know.

jamescrosswell commented 3 months ago

Very interesting read. That would mean that those issues that gets logged to Sentry aren't necessarily native crashes at all. Good to know.

Indeed - and I think the team that maintains sentry-native will be looking into a fix for this (so that only the managed warning is raised in these cases).

jamescrosswell commented 3 months ago

Should be addressed by https://github.com/getsentry/sentry-native/issues/1026 once this is released (and we bump to whatever version it's released in).

bruno-garcia commented 1 month ago

Looks like for us to be able to move forward with:

https://github.com/getsentry/sentry-native/pull/1027

We need to have that alpha release brought into the .NET SDK, bindings adjusted so we can call that method @markushi talked about on his comment:

https://github.com/getsentry/sentry-native/pull/1027#issuecomment-2396006412

And with that make a -alpha release of the .NET SDK so folks can try that out.

@bricefriha could you please give this a shot once you're back from the conference?

bricefriha commented 1 month ago

@bricefriha could you please give this a shot once you're back from the conference?

sure thing!

jamescrosswell commented 2 weeks ago

@espenrl sorry I realise it's been a while, but prerelease version 4.12.1-segv1 (which you can get from nuget.org) should fix this.

Is there any chance you could try this out to ensure it resolves the problem for you, before we GA these changes across the various SDKs involved?

espenrl commented 2 weeks ago

I'll look into it one of the next days. Our app has been locked at MAUI.iOS 17.5 version range creating problems like #3648 I'll first have to check if the MAUI team has solved the issues hindering us upgrading to the XCode 16 tooling - which again allows us to use Sentry 4.12 and later.

espenrl commented 1 week ago

I've tested it by forcing a native crash using SentrySdk.CauseCrash(CrashType.Native). This crashes the app instantly. My findings are:

creates a single event SIGSEGV: Segfault
seems like event is transferred to Sentry Cloud at next startup
logcat.log is not attached to the event
.NET things are not included (data for assemblies and contexts Culture, Dynamic Code, Memory Info, Thread Pool info, App, Runtime)
context Trace Details not included
additional context os_linux

4.12.1-segv1 on a Samsung 22+ running Android 14. App published the normal way through Google Play internal track.

espenrl commented 1 week ago

@jamescrosswell See my comment above, and please tell me if there is something specific you want me to test for.

bricefriha commented 1 week ago

@espenrl, thanks. Could you test with the other crash type as well? There is another issue (#3545) that prevents any attachment on the native crashes' event (we are working on that too)

espenrl commented 6 days ago

@bricefriha I've tested CauseCrash with both CrashType Managed and ManagedBackgroundThread in the unhandled case - crashing the app immediately. They both include logcat.log as an attachment.

bricefriha commented 6 days ago

Thank you so much @espenrl!

For the CrashType.Native issue, I would like to redirect it to #3545 as this is not an NDK Handler-related issue, but we are working on it

espenrl commented 6 days ago

Thank you so much @espenrl!

For the CrashType.Native issue, I would like to redirect it to #3545 as this is not an NDK Handler-related issue, but we are working on it

I'm happy to help :) And most of all I'm impressed how seriously you guys are taking these issues around logcat attachments and double logging native crashes. It makes me feel even more safe and sure about using Sentry. I also like the technical aspect of the solution how it is needed to replace the program exception handler at the native side of things. Interesting to follow :)

getsentry / sentry-dotnet