tikv / pprof-rs

A Rust CPU profiler implemented with the help of backtrace-rs
Apache License 2.0
1.26k stars 94 forks source link

SIGABORT when profiling with pyroscope-rs #219

Open alindima opened 11 months ago

alindima commented 11 months ago

I'm trying to run polkadot under pyroscope-rs, which uses pprof-rs as a backend. Running it on MacOS is impossible, as I run into EXC_BAD_ACCESS.

Running on Linux also fails, with SIGABRT. Here's the stack trace:

#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140089216644672) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140089216644672) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140089216644672, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007fcb25242476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007fcb252287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007fcb255c76b1 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#6  0x00007fcb255db915 in _Unwind_Backtrace () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#7  0x0000558d48284df9 in backtrace::backtrace::libunwind::trace (cb=...) at /home/alin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.67/src/backtrace/libunwind.rs:93
#8  backtrace::backtrace::trace_unsynchronized<pprof::profiler::perf_signal_handler::{closure_env#0}> (cb=...) at /home/alin/.cargo/registry/src/index.crates.io-6f17d22bba15001f/backtrace-0.3.67/src/backtrace/mod.rs:66
#9  0x0000558d4828d011 in pprof::backtrace::backtrace_rs::{impl#1}::trace<pprof::profiler::perf_signal_handler::{closure_env#0}> (cb=...) at src/backtrace/backtrace_rs.rs:23
#10 0x0000558d48285ef2 in pprof::profiler::perf_signal_handler (_signal=27, _siginfo=0x7f690ffc9f30, ucontext=0x7f690ffc9e00) at src/profiler.rs:336
#11 <signal handler called>

I can supply the coredump if needed for debugging purposes.

This renders this crate and pyroscope completely unusable for our purposes, so please advise.

alindima commented 11 months ago

I cherry-picked this fix and still getting the same result