tikv / pprof-rs

A Rust CPU profiler implemented with the help of backtrace-rs
Apache License 2.0
1.29k stars 99 forks source link

Sampling frequency cannot be guaranteed #177

Open viglia opened 1 year ago

viglia commented 1 year ago

Hi,

I don't know if this issue should be labelled as a "bug" since it's due to the way sampling is implemented but we probably should make clear in the docs that the sampling frequency is "indicative" and not real.

From the README:

When profiling was started, setitimer system call was used to 
set up a timer which will send a SIGPROF to this program 
every constant interval.

When receiving a SIGPROF signal, the signal handler will capture 
a backtrace and increase the count of it. 

etc. etc.

If we check the Execution of signal handlers section of the Linux Manual Page though, it explains that:

   Execution of signal handlers
       Whenever there is a transition from kernel-mode to user-mode
       execution (e.g., on return from a system call or scheduling of a
       thread onto the CPU), the kernel checks whether there is a
       pending unblocked signal for which the process has established a
       signal handler.  If there is such a pending signal, the following
       steps occur:

This has 2 consequences:

  1. we cannot guarantee when and how often a sample is collected
  2. depending on the nature of the work being executed in a thread, sampling might be biased toward certain threads at the expense of others

In my experiments I have indeed come across the 2 points I've mentioned above.