search

fluxcd / flux2-monitoring-example

Prometheus monitoring for the Flux control plane
https://fluxcd.io/flux/monitoring/
Apache License 2.0
52 stars 145 forks source link

Grafana Dashboards: displays NaN as a result for queries that runs division operation #22

Open antonblr opened 10 months ago

antonblr commented 10 months ago

Flux Cluster stats dashboard (haven't tried the other one yet) shows NaN in places where it runs queries with division operation, .e.g.

sum(rate(gotk_reconcile_duration_seconds_sum{namespace=~"$operator_namespace",exported_namespace=~"$namespace",kind=~"Kustomization|HelmRelease"}[5m])) by (kind)
/ sum(rate(gotk_reconcile_duration_seconds_count{namespace=~"$operator_namespace",exported_namespace=~"$namespace",kind=~"Kustomization|HelmRelease"}[5m])) by (kind)

image

image

This due to division by zero. My workaround is to add > 0 to such queries (I added it everywhere, but it can be limited to divisor only), like:

sum(rate(gotk_reconcile_duration_seconds_sum{namespace=~"$operator_namespace",exported_namespace=~"$namespace",kind=~"Kustomization|HelmRelease"}[5m]) > 0) by (kind)
/ sum(rate(gotk_reconcile_duration_seconds_count{namespace=~"$operator_namespace",exported_namespace=~"$namespace",kind=~"Kustomization|HelmRelease"}[5m]) > 0) by (kind)