tikv / pprof-rs

A Rust CPU profiler implemented with the help of backtrace-rs
Apache License 2.0
1.3k stars 99 forks source link

Trace/BPT trap on Mac #131

Open DavidBJaffe opened 2 years ago

DavidBJaffe commented 2 years ago

This code:

use pprof::ProfilerGuard;
static mut GUARD: Option<ProfilerGuard<'static>> = None;
static mut BLACKLIST: Vec<String> = Vec::new();
pub fn start_profilingx(blacklist: &[String]) {
    let frequency = 1000;
    unsafe {
        BLACKLIST = blacklist.to_vec();
        GUARD = Some(pprof::ProfilerGuard::new(frequency).unwrap());
    }
}
pub fn profiling_blacklistx() -> Vec<String> {
    vec!["woof".to_string()]
}
fn main() {
    start_profilingx(&profiling_blacklistx());
    let mut mc = 0;
    for i in 0..10_000 {
        for j in 0..10_000 {
            let n = i * j;
            mc = (mc + n) % 999;
        }
    }
    eprintln!("mc = {}", mc);
}

yields Trace/BPT trap when I run it, at least most of the time. Other similar code does this most of the time.

It is exceptionally finicky. For example, changing vec!["woof".to_string()] to vec![] eliminates the problem.

This is using master on pprof, but also occurs in 0.6.

I believe that the problem may be restricted to particular Mac versions. I have:

rustc 1.60.0 OSX 12.4 chip = apple M1 Pro

[profile.dev] debug = 1 opt-level = 3 split-debuginfo = "unpacked"

DavidBJaffe commented 2 years ago

I have now encountered the same error when not using pprof at all.

DavidBJaffe commented 2 years ago

Reopening. The problem is presumably with libunwind, but it makes pprof unusable on a Mac, or more precisely on my Mac. Another person reproduced the problem on a non-M1 Mac. Maybe the problem occurs on all Macs.

ricardopieper commented 2 years ago

I'm having this problem as well, on a Mac M1 pro, same thing as yours, no profile.dev settings. Throws somewhere on libunwind when calling .pprof() at this part:

libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::step() (@libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::step():162)

DavidBJaffe commented 2 years ago

I've actually been unable to get any form of profiling to work on my Mac, so for me at least, getting this to work would be phenomenally helpful.

YangKeao commented 2 years ago

For mac user, the frame-pointer is a more suggested way to get the backtrace, as the mac os has established an ecosystem of using the frame pointer, which is much more efficient and simpler than dwarf. More information can be found in https://github.com/tikv/pprof-rs/issues/75#issuecomment-1105687213 .

In the latest nightly of pprof-rs, enabling the frame-pointer feature will use the frame-pointer to get the backtrace. For the 0.9 release, you could disable the backtrace-rs feature (which is the default) and enable the frame-pointer feature.

If you insist on using the dwarf way, you could try to add more things to the blacklist (according to the fault backtrace). It seems that some parts of mac os (context.pc) don't deliver with understandable dwarf information. We have experience the same situation on https://github.com/tikv/tikv/issues/9957#issuecomment-813368686 .

DavidBJaffe commented 1 year ago

Yes, that works really well, thank you!