Automated crash analysis

[gsingh93]

I have some scripts that I've been using for a few weeks to automate some parts of figuring out why a binary crashed, in order to decrease debugging time and speed up exploitation. It would be good to have some similar things natively in pwndbg.

The full automation I have involves watching a directory for coredumps, and soon as any appear a script will trigger the crash analysis and notify me with any useful information, but this issue is only concerning creating some command that will provide that useful information.

This command should work both when a process crashes while running in GDB (and maybe even run automatically, if configured?), and on coredump files. People will probably have different ideas for what useful information we could surface, but here are a few starting points:

• Output the fault_addr, and check any registers match this fault address.
• Check for memory dereference registers in the faulting instruction and specifically show those to the user. Maybe do some special handling if it's the frame pointer that's causing the fault to help figure out how it happened.
• Warn the user when rip, rbp, rsp, etc. point outside of the expected mappings
• If the text in rip looks like the text from cyclic, automatically do cyclic_find on it and show the offset
• If the stack pointer is not aligned and the faulting instruction is using xmm registers, explicitly point this out to the user ("Crash occurred due to unaligned stack pointer).
• Potentially do some automated analysis when __stack_chk_failed is in the backtrace, showing what the canary got set to, and what the surrounding values on the stack are.
• Detect when we've aborted due to a failed glibc heap corruption check, and then show the user the specifics around why the check failed.
• Sometimes GDB's disassembly doesn't work because rip gets corrupted. We should walk up the stack frame and output the disassembly in the calling frame.

The starting point for this would be to implement some command (i.e. analyze-crash) that implements at least one of the ideas above, or some other idea, and then we can go from there.

[0xricksanchez]

Sounds like this would go hand in hand with forking and updating/extending exploitable with an integration step right into pwndbg.

Edit: That said automatic crash analysis (aka root cause analysis) that goes beyond looking at the latest stack frame and what $rip contains is far from trivial. If you want to follow that rabbit hole:

• Igor: Crash Deduplication Through Root-Cause Clustering
• AURORA: Statistical Crash Analysis for Automated Root Cause Explanation
• Linux crash utility
• Analysis and Visualization of Linux Core Dumps