Autogenerate a "real" set of rules files consumable by prometheus

[robert-will-brown]

Copy and pasting these rules is ok, but it would be easier if there was a generated set of rules files from rules.yml that would "just work".

Is there any reason this isn't generated? If not are there any templating tools that anyone knows of that could achieve this?

[paulfantom]

A lot of alerting rules here are taken from monitoring mixin projects (which produce "real" rule files). For more, you can see for example following sub-projects:

• https://github.com/prometheus/prometheus/tree/master/documentation/prometheus-mixin
• https://github.com/kubernetes-monitoring/kubernetes-mixin
• https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin
• https://github.com/etcd-io/etcd/tree/master/Documentation/etcd-mixin

Mixins are what prometheus community agreed on to provide a customizable package containing recording rules, alerts, and grafana dashboards. For more info on mixins you can read in https://github.com/monitoring-mixins/docs

And mixins can be combined together and inherited. One example of more advanced use-case could be seen in https://github.com/coreos/kube-prometheus project.

[robert-will-brown]

Thanks for the info, I’ll have a look.

[robert-will-brown]

Hey @paulfantom, so I've looked at these and are non the wiser for it! Are you suggesting that it would be a good idea or bad idea to also produce some immediately useable templates?

If good idea, then I can't immediately see how these mixins help get the rules.yml file into a usable collection of prometheus consumable YAML files?

[paulfantom]

Are you suggesting that it would be a good idea or bad idea to also produce some immediately useable templates?

Mixins are written in jsonnet language, which allows overriding any parameter, so I don't see any benefit of using templating. It isn't the easiest path to follow but provides the best customizability options, which is quite important when someone creates a library (which mixins classify as).

If good idea, then I can't immediately see how these mixins help get the rules.yml file into a usable collection of prometheus consumable YAML files?

If you mean rules.yml file from this repository, then they don't. I was pointing you to an already established "standard" of sharing alerts, recording rules, and dashboards instead of inventing "brand new thing" and rewriting this repository. And if you want to create "brand new thing", please read design doc of monitoring-mixins and thing what would be the maintenance toll if templating was introduced in this repository.

---

In the end it is @samber decision if he wants this repository to be a guideline (as it is now) or a library (if templating was introduced). :)

[b-a-t]

I think this issue deserves to be in the README.md or around. I also was wondering why even code snippets are not directly usable in the configs and now I see why. But how many still do have this question?

[samber]

You can run wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/master/dist/rules/kafka/danielqsj-kafka-exporter.yml.

Anyway, a better tooling should be used when available.

[sjentzsch]

@samber I also stumbled upon this: So far I saw awesome-prometheus-alerts as a collection of links to various mixins, however, it became more; we somewhat re-do the work. For example, the node-exporter alerts within this repo have nothing to do with the node-exporter mixin alerts (https://github.com/prometheus/node_exporter/blob/master/docs/node-mixin/alerts/alerts.libsonnet). Yes, they are related, probably inspired, maybe derived from the mixin, they are partly even much better, but they are completely different (starting with the naming already).

As a user, I wonder what awesome-prometheus-alerts will become. Even so, to which project I should commit my shiny-and-great alert that I have for a component: This repo, or the related upstream-mixin? Do they complement each other, do they compete against each other? Now I would always compare, e.g. for Thanos: What has the upstream mixin to offer, what is in this repo, render both to manifests, run a diff - and then basically merge manually.

No criticism at all, but maybe indeed it would be good to have some statements on the philosophy of this repo right inside the README.

[dudicoco]

I agree with @sjentzsch.

Rather than diverging from the upstream mixin projects it would be much better if this project would just unify all of these projects in one place without altering them and add any other missing mixins for different categories (redis, kafka etc.) which might not have existing mixin projects.

Enhancements should be contributed to the upstream projects so we as the users can enjoy well maintained sets of rules.

The way this can be achieved is by setting up a sync job which fetches upstream changes daily (I believe the prometheus-operator project does that) and making sure these changes are not modified.