search

EmbarkStudios / crash-handling

Collection of crates to deal with crashes
Apache License 2.0
138 stars 13 forks source link

Segfault in `minidump_writer::linux::thread_list_stream::write` when `opt-level` > 1 #84

Closed timfish closed 7 months ago

timfish commented 7 months ago

I recently released sentry-node-minidump which uses napi-rs to expose some of the code I wrote in minidumper-child to node.js to send minidumps to Sentry for node.js crashes. minidumper-child takes the code from the minidumper diskwrite example and packages it with a simpler API.

It wasn't until I actually added e2e tests in CI that I noticed that it didn't work on Linux!

After much debugging I found that the server was segfaulting when trying to write the dump file. It would write 236 bytes (presumably the header) before segfaulting in minidump_writer::linux::thread_list_stream::write. I noticed that this wasn't the case in debug mode, so I've managed to work around this by overriding opt-level = 1 for the Linux release build. opt-level = 2 results in the crash. This isn't actually a huge issue for sentry-node-minidump since I build and distribute the binaries.

It was at this point I realised that the CI tests for minidumper-child are only testing the debug profile. I opened a PR to test the release profile and it's failing, presumably for the same reason. This suggests the issue has nothing to do with node.js or napi-rs since that code is all Rust: https://github.com/timfish/minidumper-child/pull/2

Should I have opened this issue in minidump-writer?

Have you seen anything like this before?

Jake-Shadle commented 7 months ago

So I had a mild panic attack last night going to bed because I foolishly read this issue, but I am able to repro and it's not as bad as I had feared. This is a regression from 1.77.0 (so it's never happened before), and is due to the changes in u128 layout described here. I'm doing a fix now, thank you for reporting!

Jake-Shadle commented 7 months ago

This was released in 0.8.9 of minidump-writer.

timfish commented 7 months ago

Ah perfect, thanks for the quick fix.

I didn't even think of trying previous Rust versions!