observability: integrate CPU & memory profiling with Grafana Pyroscope

risingwavelabs / risingwave

SQL stream processing, analytics, and management. We decouple storage and compute to offer instant failover, dynamic scaling, speedy bootstrapping, and efficient joins.

https://www.risingwave.com/slack

Apache License 2.0

6.59k stars 538 forks source link

observability: integrate CPU & memory profiling with Grafana Pyroscope #13026

Open BugenZhao opened 8 months ago

BugenZhao commented 8 months ago

Proposed by @arkbriar.

Grafana Pyroscope is an open source software project for aggregating continuous profiling data. Grafana Pyroscope is fully integrated with Grafana allowing you to correlate with other observability signals, like metrics, logs, and traces.

Its Rust client has built-in integration with pprof-rs. We may investigate if it's possible to integrate with jeprof as well.

arkbriar commented 8 months ago

I'm not sure but I think it might also be possible to integrate the jemalloc profiling 🤔

fuyufjh commented 7 months ago

In addition, how about integrating await tree as a backend? It might be more helpful than CPU profiling especially for IO-intensive cases.

https://github.com/grafana/pyroscope-rs/blob/main/pyroscope_backends/pyroscope_pprofrs/src/lib.rs#L158

The core part is the report() function, which looks easy for us to implement one for await tree

BugenZhao commented 7 months ago

After some investigation, I'm afraid await-tree is not that compatible with Pyroscope:

The backends of Pyroscope should be sample-based. However, await-tree is not designed to be frequently captured as it'll acquire locks.
await-tree yields trees, instead of stacks. Aggregating the results may not be feasible.
await-tree does not output in a structured format now.

github-actions[bot] commented 3 weeks ago

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.