search

istio / istio.io

Source for the istio.io site
https://istio.io/
Apache License 2.0
763 stars 1.53k forks source link

Add note about scrape intervals vs. rate ranges to grafana dashboard docs #8252

Open douglas-reid opened 3 years ago

douglas-reid commented 3 years ago

We should add documentation around the interplay of scrape intervals and rate ranges to our dashboard docs.

In helping troubleshoot dashboard issues today, we stumbled upon a "gotcha" of sorts with the dashboards / Prometheus. It looks like the default helm installs of Prometheus ship with a 1m scrape interval. This is problematic, as our dashboards use a default rate range of 1m. This means that the grafana queries into Prometheus will return no data (as rate requires two data points). It looks like the general recommendation for rate range configuration is 4x scrape intervals.

At the least, we should have a warning in the docs about this very issue (and probably also in the appropriate FAQ).

It would be ideal to allow dashboard wide configuration for rate intervals. Grafana exposes the interval as a param, and that could be useful to adopt.

/cc @ldemailly @howardjohn

ldemailly commented 3 years ago

for people installing prometheus with helm,

     --set server.global.scrape_interval=15s

makes both the kiali 1min and most grafana istio dashboards work (compared to the 1min default of prom)

this being said 15s scrape is kinda frequent so indeed it'd be ideal that grafana dashboards adapts to scrape rate rather than the other way around (given prom doesn't consider giving "no data" a bug https://github.com/prometheus/prometheus/issues/8001 )