search

grafana / pyroscope-rs

Pyroscope Profiler for Rust. Profile your Rust applications.
Apache License 2.0
132 stars 22 forks source link

pyroscope-rs feedback #127

Open korniltsev opened 11 months ago

korniltsev commented 11 months ago

There have been a bunch of feedback on pyroscope-rs. It may worth to investigate. specifically mem leak and signal handlers overhead.

https://github.com/paritytech/polkadot-sdk/issues/804

  • pprof-rs has a high overhead, since it's doing profiling being based on signal handlers (and pyroscope unregisters/registers the signal handlers every 10 seconds).
  • libunwind mechanism is buggy and causes SIGABORTs: SIGABORT when profiling with pyroscope-rs tikv/pprof-rs#219
  • pyroscope-rs repo does not seem under a lot of active maintenance. I also discovered a bug there: fix spurious exit when epoll_wait is interrupted by a signal grafana/pyroscope-rs#125
  • even with frame pointer unwinding and the above fix, after a couple of hours of running the validator with pyroscope enabled, most of the nodes tasks are killed, including the pyroscope agent (I assume there's a memory leak that triggers an OOM somewhere), which leaves the node in an inconsistent but still running state.
ttys3 commented 2 months ago

high memory usage https://github.com/tikv/pprof-rs/issues/244

before usage of pyroscope-rs: 6M after usage of pyroscope-rs: 200M+