axw
commented
6 years ago I've been wanting profiling storage since forever. Now that I'm at Elastic, I thought I'd actually do something about it. I've started on a new Beat which gathers pprof profiles: https://github.com/axw/pprofbeat.
(This is my first real foray into ES and Beats, so the data format is probably not great - but it's something.)
I care mostly about profiling Go programs, which is pretty straightforward: import a standard package, and you can expose CPU and heap profile data via an HTTP endpoint. What I want to see is:
- periodic profiling (done by pprofbeat; other beats would be required for on-machine profiling, i.e. "perf" and friends)
- Kibana visualisations, with the ability to compare profiles (e.g. to see how much memory has grown over time, or how the CPU profile looks at various load levels)
I was intending to look at embedding the "pprof" web UI as a plugin, although I don't really know where to start. Needless to say, the more integrated the better.
A few screenshots of the pprof web UI below in case folks are not aware of it - might serve as inspiration.
Example pprof graph:
Pprof flamegraph:
Pprof top offenders:
Pprof source code annotation:
pmoust
formgeist
monfera
timductive
Flame graphs can be useful for visualizing output of a profiling tool: http://www.brendangregg.com/flamegraphs.html https://github.com/brendangregg/FlameGraph
I remember chatting with @danielmitterdorfer about this a while back. Would be interesting to think through how profiling data can be stored in Elasticsearch and visualized in a native Kibana visualization like this.
cc: @alexfrancoeur @thomasneirynck