grafana / grafana

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
https://grafana.com
GNU Affero General Public License v3.0
61.61k stars 11.76k forks source link

FlameGraph: The displayed samples are wrong #88847

Open simonswine opened 1 month ago

simonswine commented 1 month ago

We currently base the displayed samples on the total sample values, rather than the sample count.

That leads to wrong values being displayed. An example is this:

image

As this is a Golang CPU profiler, the standard sampling rate is 100 samples per seconds, which would mean that is the values we would expect:

1.01 seconds / 101 samples

There is more detail in the Pyroscope issue https://github.com/grafana/pyroscope/issues/3340, but it would be good to respect sampleRate in the metadata of the /render endpoint:

simonswine commented 1 month ago

So we decided to go with a solution that accepts a factor in a field called sampleRate. In case this field is missing or 0, we should no longer show the "x samples" line in the tool tip or the "| x samples" in the top field.