jan-auer
commented
4 years ago Hi @milianw. Thanks for this suggestion and the detailed context you're providing. I believe that what you're looking for can already be achieved today -- although in a slightly altered form.
Immediate Solution
Besides Minidump files, Sentry supports uploading errors and crashes in our JSON event payload format. This schema has the highest stability guarantees at Sentry, and we retain backwards compatibility. This is used by all our SDKs, including native SDKs that perform on-device stack walking, such as the Cocoa and Android (NDK) SDKs.
You can read the output of minidump_stackwalk and then directly convert it into the schema expected by our Servers without additional processing steps. The mapping is roughly:
- Crash time:
timestamp - Crash reason and other crash info:
exception.value - Threads and frames:
threadsandthread.stacktrace, particularly:- Thread id:
thread.id - Crashed thread: mark
thread.crashed - Registers:
thread.stacktrace.registers - Instruction pointer:
frame.instruction_addr
- Thread id:
- Loaded modules:
debug_meta.images - OS and CPU info:
contexts
This allows you to send a prewalked stacktrace to Sentry for symbolication, and attach additional information such as breadcrumbs, tags, or the release version. Please let us know if you have questions about the schema or if we could help with the mapping.
Click to expand example payload
```json { "timestamp": 1521713273, "contexts": { "os": { "name": "Windows", "version": "10.0.14393" }, "device": { "arch": "x86" } }, "exception": { "type": "EXCEPTION_ACCESS_VIOLATION_WRITE", "value": "Assertion Error: foo" }, "threads": [ { "crashed": true, "id": 1636, "registers": { "eax": "0x0", "ebp": "0x10ff670", "ebx": "0xfe5000", "ecx": "0x10ff670", "edi": "0x13bfd78", "edx": "0x7", "eflags": "0x10246", "eip": "0x2a2a3d", "esi": "0x759c6314", "esp": "0x10ff644" }, "stacktrace": { "frames": [ {"instruction_addr": "0x2a2a3d"}, {"instruction_addr": "0x2a28d0"}, {"instruction_addr": "0x7584e9bf"} ] } }, { "id": 3580, "registers": { "eax": "0x0", "ebp": "0x159faa4", "ebx": "0x13b0990", "ecx": "0x0", "edi": "0x13b4af0", "edx": "0x0", "eflags": "0x216", "eip": "0x771e016c", "esi": "0x13b4930", "esp": "0x159f900" }, "stacktrace": { "frames": [ {"instruction_addr": "0x771e0160"} ] } } ], "debug_meta": { "images": [ { "type": "pe", "code_id": "5ab380779000", "code_file": "C:\\projects\\crash.exe", "debug_id": "3249d99d-0c40-4931-8610-f4e4fb0b6936-1", "debug_file": "C:\\projects\\crash.pdb", "image_addr": "0x2a0000", "image_size": 36864 }, { "type": "pe", "code_id": "5a49bb75c1000", "code_file": "C:\\Windows\\System32\\rpcrt4.dll", "debug_id": "ae131c67-27a7-4fa1-9916-b5a4aef41190-1", "debug_file": "wrpcrt4.pdb", "image_addr": "0x75810000", "image_size": 790528 }, { "type": "pe", "code_id": "59b0d8f3183000", "code_file": "C:\\Windows\\System32\\ntdll.dll", "debug_id": "971f98e5-ce60-41ff-b2d7-235bbeb34578-1", "debug_file": "wntdll.pdb", "image_addr": "0x77170000", "image_size": 1585152 } ] } } ```Supporting minidump_stackwalk
There is no technical reason against adding support for the output format of minidump_stackwalk. However, we believe in portable and reusable APIs for error and crash ingestion. Our event schema serves a multitude of platforms outside of native crashes, like Python or JavaScript. For this reason, in most cases we gravitate to evolve the existing protocol rather than adding new formats.
We do adopt widely spread formats that are otherwise hard to convert. As part of that, our ingestion pipeline supports Minidumps, UE4 crash archives, and Apple crash reports. I think that we can evaluate adding the minidump_stackwalk output format in the future, but at the moment it looks like there are viable alteratives to that.
System Symbols
I agree, on-device stackwalking has many advantages particularly when third-party symbols and system symbols are not available. For macOS, iOS and Windows, we maintain a comprehensive repository of system libraries particularly for stackwalking and basic symbolication (using symbol tables only), and we integrate with third parties such as the Microsoft Symbol Server.
Unfortunately, this falls short on Linux and Android. There have been recent advancements to introduce symbol servers for Linux, such as debuginfod, but we still have a long way to go to. On Android, fragmentation makes matters much worse and on-device stackwalking remains the only reliable strategy to achieve proper stack traces.
Stackwalking and Symbolication
I can currently see a few major challenges with running minidump_stackwalk on-device:
- Breakpad only understands its own symbol format, and
dump_symscan only be executed on the same platform. For example, a PDB can only be converted into Breakpad symbols usingdump_syms.exeand not the Linux build. - To my understanding, the basic
minidump_stackwalkutility expects symbols in a special directory structure. An additional preparation step is needed to collect symbols into those folders. Of course, with Sentry it is similar, as one has to usesentry-clito upload all files. - For split debug files, the part containing full inline information, function names, and file paths, is the much bigger part -- usually 10x the size of an optimized stripped binary. Sentry still needs this file if you want it to symbolicate, so the savings are limited.
- Of course, you can also run all symbolication on the device, but that will have both performance and storage impact. I'm assuming that you are primarily targeting server applications. For applications shipped to consumer devices, packaging debug information or Breakpad files, and minidump_stackwalk is most likely a red line.
Status of Crash Support
Sadly, the current status of sentry's native crash support makes it often impossible to investigate crash reports, as the stacks are completely bogus.
This is a very valid point, and thank you for bringing this up. While we continue to integrate more symbol servers and supply unwind information for Linux and Android, there will always be missing symbols. With FPO and heavy optimization, there is no guarantee we can create sane stack traces in those cases.
There are a few things on our mind on how we are going to address this:
- Allow to reprocess a crash or an issue after supplying the missing files. With this, the stack trace is not lost and can be recovered as soon as the missing files have been uploaded to Sentry or a symbol server.
- Use a prewalked stack trace attached to the Minidump. Currently, Sentry only reads stack traces from minidumps. Some crash reports, like the UnrealEngine 4 Crash Reporter, already supply a stack trace along with the minidump. Sentry could use such a trace as a fallback when symbols are missing.
- Move to more on-device stack walking in our SDKs. Likewise, our SDKs rely on Breakpad and Crashpad to do the heavy lifting on all major desktop platforms. Instead, we can ship unwinders with the SDK and then send a JSON payload for symbolication like in our iOS SDK.
Thank you so much for offering your help. What do you think about the above, and what can we do to improve the service?
milianw
Summary
In addition to uploading raw minidump files, it would be ideal if pre-processed raw stack traces could be uploaded instead.
Motivation
The main problem with the current support for native crash dumps is that it requires uploading of each and every DSO that's encountered in any frame of the stack trace. As soon as this isn't the case, the really unreliable stack scanning approach is used, which is usually leading to totally bogus stack traces.
If the minidump files would be unwound locally (cf.
minidump_stackwalk) and then the raw stack trace uploaded to sentry, the stack trace should be correct. Sentry would then only be responsible for the symbolication.This allows another potential improvement: One doesn't need to upload the full (large) debug DSO files to sentry. Instead, it would be enough to upload the (compressed) output of
dump_syms. That would solve a second problem we are seeing - namely the immense amount of disk space required to host all of the debug symbols to support multiple versions of our large native application.Additional Context
Take a typical Linux application that's provided as an AppImage: One would not bundle the libc.so within that AppImage. If we hit a crash in libc.so (e.g. invalid
freeor similar), then sentry would fail to create a proper backtrace, unless one uploads the libc.so from all supported linux distributions.If instead the unwinding is done locally, all DSOs are available and the
.eh_framesection can be used to create a fully unwound stack. This would only contain instruction pointer addresses (IPs). Together with the memory map which is included in the output ofminidump_stackwalk, sentry can then map the absolute IPs into relative DSO offsets and lookup the correctdump_symsfile with the matching build-id to find the symbol name, and potentially add inline frames too.If you have any questions to this, I'm willing to help you out. As the author of heaptrack and hotspot, I've learned a few things about how unwinding and symbolication works. Sadly, the current status of sentry's native crash support makes it often impossible to investigate crash reports, as the stacks are completely bogus. The suggestion above should alleviate the problem.
Otherwise: Thanks a lot for creating Sentry - it has a lot of potential!