Create a default set of prometheus alerts

[costinm]

@nmnellis commented on Wed Oct 18 2017

FEATURE REQUEST Describe the feature:

For users that have Prometheus installed, document a set of alerts that an istio user might find useful after istio is running. Some metrics may need to be created to effectively monitor istio.

A short list of nice to have monitors:

• High response times between two http services
• Number of 4xx or 5xx errors abnormally high
• Increase in response volume

There are more but this is a start

---

@douglas-reid commented on Thu Oct 19 2017

@nmnellis Great idea! I think there is lots of ripe territory for improving the observability story here. Perhaps we should flesh out starting points in a document (for instance, should we also use Grafana alerting, etc.?), and then move that into actionable work items. Would you be interested in helping to start such a document (or help me in drafting one)?

[nmnellis]

@douglas-reid yeah ill start on a Document. Is there any documentation about the existing metrics that are exposed to grafana? If not i think it would also be beneficial to list the metrics listed as well.

[douglas-reid]

@nmnellis Awesome! Please let me know when you've started and I will hop in to help contribute (if you don't mind).

There isn't any documentation on the metrics, other than the config for them that exists in the various istio/istio yaml files (which are built from: istio-mixer.yaml.tmpl). Look for anything with a kind:metric (example: requestcount).

One note on "existing" metrics: Istio does not hard-code or require any of these metrics. Any operator is free to change their shape. We provide a set by default that is paired with our example dashboard, but both of those are editable.

One of the things I want to work on (or perhaps find a motivated contributor <wink, wink> to work on ) is auto-generating the dashboard based on known services and metrics, via something like grafanalib. Doing that + generating alerts would be even slicker.

[nmnellis]

@douglas-reid I started looking at the default metrics you have configured and i noticed that they are not prefixed in prometheus like other alerts. example request_count. it would be nice to make the defaults namespaced similar to what kube-state-metrics does (by appending kube-) to all its metrics. is this as simple as changing the metrics name? Edit: submitted a PR as an example: https://github.com/istio/istio/pull/1212

[nmnellis]

Also is there documentation about the data envoy provides to mixer on each call?

[douglas-reid]

@nmnellis please see https://istio.io/docs/reference/config/mixer/attribute-vocabulary.html for information on attributes available on Report calls.

[douglas-reid]

update: i've suggested the approach of adding the namespace directly in the prometheus adapter. i will be generating the PRs necessary for that in the coming days, as I work with @nmnellis to put together initial design for alertmanager configs in prometheus for istio.

[nmnellis]

@douglas-reid i've started a document in my own fork for metrics and alerting. it's a mess currently but if there is a better way to share it let me know. maybe a google doc somewhere? https://github.com/nmnellis/mixer/blob/32dab1e0d553692be05496d6e3a6abd2e742da16/adapter/prometheus/alerts.md