search

linkerd / linkerd-viz

Top-line service metrics dashboard for Linkerd 1.
Apache License 2.0
111 stars 32 forks source link

Configurable scrape_interval is broken, breaks 1m Grafana graphs #34

Closed pforman closed 7 years ago

pforman commented 7 years ago

The changes in #33 made for a surprise, when the success rate and request volume graphs became empty.

The default of 1m matches the query in Grafana, so the graphs become empty when they don't have two data points. I'd recommend a different default, like 30s.

However, this can't be overridden, because the replacement with $SCRAPE_INTERVAL fails.

The root of the issue is here:

sed -i "" "s@scrape_interval:.*@scrape_interval: $SCRAPE_INTERVAL@" $PROMETHEUS_CONF
sed -i "" "s@ evaluation_interval:.*@ evaluation_interval: $SCRAPE_INTERVAL@" $PROMETHEUS_CONF

With sed -i "" , the "" is interpreted as a filename by GNU sed, and the sed command fails. So the replacements never happen, and the command is only ever executed with the default "1m" in prometheus-$PLATFORM.yml . (whichever file that turns out to be, "k8s" in my case).

This should be sed -i"", with no space. (verified inside the container)

It would be a two-character PR, but the default causing empty graphs is also surprise, so the defaults should probably be adjusted down to ensure the irate() call has data.

wmorgan commented 7 years ago

@pforman thanks for the detailed analysis! We will take a look.

siggy commented 7 years ago

@pforman Thanks for the report! Fix is up at #35.