Creating events that get symbolicated using the SDK

[bruno-garcia]

Using the following code:

// Create exception object
sentry_value_t exception = sentry_value_new_exception("Crash", "Fatal Error: EXC_ARITHMETIC");

// Create stack frames
sentry_value_t frames = sentry_value_new_list();

sentry_value_t frame1 = sentry_value_new_object();
sentry_value_set_by_key(frame1, "function", sentry_value_new_string("start"));
sentry_value_set_by_key(frame1, "instruction_addr", sentry_value_new_string("0x7ff810252344"));
sentry_value_set_by_key(frame1, "package", sentry_value_new_string("/usr/lib/dyld"));
//sentry_value_set_by_key(frame1, "addr_mode", sentry_value_new_string("abs"));
//sentry_value_set_by_key(frame1, "image_addr", sentry_value_new_string("0x10e531000"));
sentry_value_set_by_key(frame1, "in_app", sentry_value_new_bool(1));
sentry_value_append(frames, frame1);

sentry_value_t frame2 = sentry_value_new_object();
sentry_value_set_by_key(frame2, "function", sentry_value_new_string("main"));
sentry_value_set_by_key(frame2, "instruction_addr", sentry_value_new_string("0x00010e5327c5"));
sentry_value_set_by_key(frame2, "package", sentry_value_new_string("/Users/xxxxxx/Projects/XCodeCrashpadDemo/sentry_demo/build/Release/sentry_demo"));
//sentry_value_set_by_key(frame2, "addr_mode", sentry_value_new_string("abs"));
//sentry_value_set_by_key(frame2, "image_addr", sentry_value_new_string("0x10e531000"));
sentry_value_set_by_key(frame2, "in_app", sentry_value_new_bool(1));
sentry_value_append(frames, frame2);

sentry_value_t frame3 = sentry_value_new_object();
sentry_value_set_by_key(frame3, "function", sentry_value_new_string("causeDivisionByZero"));
sentry_value_set_by_key(frame3, "instruction_addr", sentry_value_new_string("0x00010e532403"));
sentry_value_set_by_key(frame3, "package", sentry_value_new_string("/Users/xxxxxx/Projects/XCodeCrashpadDemo/sentry_demo/build/Release/sentry_demo"));
//sentry_value_set_by_key(frame3, "addr_mode", sentry_value_new_string("abs"));
//sentry_value_set_by_key(frame3, "image_addr", sentry_value_new_string("0x10e531000"));
sentry_value_set_by_key(frame3, "in_app", sentry_value_new_bool(1));
sentry_value_append(frames, frame3);

// Create stacktrace object
sentry_value_t stacktrace = sentry_value_new_object();
sentry_value_set_by_key(stacktrace, "frames", frames);

// Attach stacktrace to exception
sentry_value_set_by_key(exception, "stacktrace", stacktrace);

// Create debug_meta
sentry_value_t debug_meta = sentry_value_new_object();
sentry_value_t images = sentry_value_new_list();
sentry_value_t image = sentry_value_new_object();

// All these values are from the JSON I pulled from Sentry via the API. The event is from a real minidump that was properly symbolicated.
sentry_value_set_by_key(image, "type", sentry_value_new_string("macho"));
sentry_value_set_by_key(image, "debug_id", sentry_value_new_string("bcfb7b82-900e-3cfa-8810-4230a3e79330"));
sentry_value_set_by_key(image, "code_id", sentry_value_new_string("bcfb7b82900e3cfa88104230a3e79330"));
sentry_value_set_by_key(image, "debug_file", sentry_value_new_string("sentry_demo"));
sentry_value_set_by_key(image, "code_file", sentry_value_new_string("/Users/xxxxxx/Projects/XCodeCrashpadDemo/sentry_demo/build/Release/sentry_demo"));
sentry_value_set_by_key(image, "image_addr", sentry_value_new_string("0x10e531000"));
sentry_value_set_by_key(image, "image_size", sentry_value_new_int32(8192));
sentry_value_set_by_key(image, "arch", sentry_value_new_string("x86_64"));
sentry_value_append(images, image);
sentry_value_set_by_key(debug_meta, "images", images);

// Create the event
sentry_value_t event = sentry_value_new_event();

// Attach exception and debug_meta to the event
sentry_value_t exception_list = sentry_value_new_list();
sentry_value_append(exception_list, exception);
sentry_value_set_by_key(event, "exception", exception_list);
sentry_value_set_by_key(event, "debug_meta", debug_meta);

// Capture the event
sentry_capture_event(event);

Events don't get symbolicated, even though the memory address used is what came out of a minidump symbolication

Note that in the case of a JSON event sent, the address links are not clickable. Perhaps it's an indication of no connection to the symbol file?

Steps taken:

1. Create a new project in Sentry, type minidump
2. Upload symbols from a crashed test app
3. Upload a minidump from a crashed test app
4. Make sure it's symbolicated properly
5. Get a json from the event
6. Use values from the json when submitting a new json event

[supervacuus]

Hi @bruno-garcia, I think this should be cross-checked with @loewenheim because I am also not 100% sure what would be required if a non-minidump event needs to be symbolicated in the backend. We typically symbolicate client-side (via the symbols table) if we send raw stack traces (although I know that I used inproc with raw stack frames in the event without client-side symbolication on Windows, so maybe that is a macOS-specific issue).

The best would be to check where the server-side symbolication fails. However, from the client side, a couple of things come to mind:

Ensure that no client-side symbolication happens

sentry_options_set_symbolize_stacktraces(options, false);

Otherwise, this would happen for all (expected) stack-frame locations during the event scope application. But I guess you're already doing that. I only wanted to ensure since it is not part of the posted code.

stack traces typically sent without meta-data

I am not sure whether any processing in the backend would take this as a hint of an already symbolicated stack frame, but I would not send the stack frames with function, package, or in_app properties, but only raw instruction_addr. You do this using sentry_value_set_stacktrace() on the exception object:

intptr_t frames[] = {
  0x00010e532403,
  0x00010e5327c5,
  0x7ff810252344,
};
sentry_value_set_stacktrace(exception, (void*)frames, sizeof(frames)/sizeof(intptr_t));

The exception interface requires a top-level values list

The way you construct it, it results in:

  "exception": [
    {
      "type": "Crash",

Whereas the docs say it should be

  "exception": {
    "values": [
      {
        "type": "Crash",

There is no need to provide debug_meta

Maybe I misunderstood the task here, and the sending application is not the one from which the stack trace is produced, but if they are the same, then the Native SDK will send a more than complete module list with each event.

Missing mechanism in the exception object

This might also not be relevant to this particular issue, but mechanism.handled, mechanism.synthetic, and mechanism.type affect how events are processed and rendered in the UI.

---

I don't see any other issues. Does this only happen when following this approach on macOS, or do you see the same issue on other platforms?

[loewenheim]

@bruno-garcia do we have a Sentry event showing this behavior?

[bruno-garcia]

I am also not 100% sure what would be required if a non-minidump event needs to be symbolicated in the backend.

We send C# and Unity errors that get symbolicated without a minidump so should work.

@bruno-garcia do we have a Sentry event showing this behavior?

I'll ask

[supervacuus]

We send C# and Unity errors that get symbolicated without a minidump so should work.

I understand it should work; as I wrote above, I had no issues symbolicating raw stack traces without client-side symbol lookup using inproc on Windows (which, besides mentioned meta-data, uses the same interface as in the code above).

My point with "not 100% sure" was that since this is not a regular use case in the Native SDK, we might be missing meta-data when constructing the module list or the exception object that previously wasn't required but now is. With the module list, it could be platform-specific (e.g., the macOS module finder needs a fix). But in that case, I would still need feedback on where and why symbolication fails.

@bruno-garcia do we have a Sentry event showing this behavior?

@loewenheim, I can reproduce this quickly if it helps speed things up. Let me know.

[loewenheim]

@supervacuus I took a look at an example pair of events (one minidump, one "recreated" from it) today. The stacktraces were identical, but the modules weren't. Is that also the case in your recreation?

[alex-gubernsky]

Hi @supervacuus, thanks for looking into this and for the tips. I'm going to try this on Windows, as well as disable client-side symbolication and send only raw instruction_addr.

The exception interface requires a top-level values list

It seems that values can be omitted according to the docs, but please let me know if that's not the case:

{
  "exception": [
    {
      "type": "Error",
      "value": "An error occurred",
      "mechanism": {
        "type": "generic",
        "handled": false
      }
    }
  ]
}

There is no need to provide debug_meta Maybe I misunderstood the task here, and the sending application is not the one from which the stack trace is produced, but if they are the same, then the Native SDK will send a more than complete module list with each event.

Correct, we are considering a scenario where an out-of-process crash handler will collect all relevant information from a crashed app and report it to Sentry. In this case, I assume debug_meta should be provided?

[supervacuus]

Hi @supervacuus, thanks for looking into this and for the tips. I'm going to try this on Windows, as well as disable client-side symbolication and send only raw instruction_addr.

Hi, @alex-gubernsky! While this was all sensible advice, thanks to your input and a short sync with @loewenheim, I can provide better feedback on the issue you are observing. You don't need to experiment with these options since this isn't an issue with the platform or stack trace metadata (although disabling client-side symbolication and only sending instruction_addr is a good idea anyway).

We looked at your event and saw that the symbolication failed because the primary module the stack frames referenced was missing (i.e., the one you manually added to debug_meta.images). This is because it will be overwritten by the module list of the current process when the event is prepared for sending.

Correct, we are considering a scenario where an out-of-process crash handler will collect all relevant information from a crashed app and report it to Sentry. In this case, I assume debug_meta should be provided?

Thanks, this is the most essential input to understand the issue. To support that use case, we would have to add an option to the SDK that turns off the application of the module list for each event. In that case, you have complete control over the images being sent.

A way to quickly verify this would be to write the debug_meta.images in the before_send hook instead of when constructing the event since the SDK invokes that hook after applying the scope to the event. Taking your example code:

sentry_value_t
add_debug_meta_before_send(sentry_value_t event, void *hint, void *closure)
{
    // Remove the one we created for the currently running process (you could, of course, filter more specifically)
    sentry_value_remove_by_key(event, "debug_meta");

    // ... your `debug_meta` construction as you did in the example ...

    sentry_value_set_by_key(event, "debug_meta", debug_meta);

    return event;
}

int main()
{
    // In your init code. The last parameter could provide access to a struct from 
    // which to retrieve debug_meta source data, it will be passed to each invocation 
    // as the `void *closure` parameter of the callback.
    sentry_options_set_before_send(opts, add_debug_meta_before_send, NULL);

    // ...

    sentry_init(opts);

    // ...

}

The exception interface requires a top-level values list

It seems that values can be omitted according to the docs, but please let me know if that's not the case:

Yes, sorry, my bad. I was surprised to find this to be an option.

[alex-gubernsky]

Hi @supervacuus, thank you for your quick response!

A way to quickly verify this would be to write the debug_meta.images in the before_send hook instead of when constructing the event since the SDK invokes that hook after applying the scope to the event.

Ah, nice! I'll give it a try and let you know how it goes.

[alex-gubernsky]

Hi @supervacuus,

A quick update: I can confirm that the reported images are no longer being overridden using the trick with the before_send callback. I can see them present in the JSON after exporting an event, and there is also a slight change in behaviour in the UI, where instruction addresses are now displayed as relative. However, that’s pretty much it. The stack frames still cannot be symbolicated, so it seems like something is still missing.

I compared the JSONs from a real minidump event and the one constructed manually, and it seems the only differences are the number of images and the number of threads. Could that be the reason?

[kahest]

@alex-gubernsky @supervacuus: fyi @loewenheim opened a ticket to investigate why "synthetic" events aren't symbolicated

[kahest]

@loewenheim afaik this has been solved, correct?

[alex-gubernsky]

Yes, this has been resolved by setting the mechanism type field to a value other than minidump.

[kahest]

Thank you for confirming @alex-gubernsky !