search

rancher / rancher

Complete container management platform
http://rancher.com
Apache License 2.0
23.45k stars 2.97k forks source link

RFE: Ability to add prometheus rules to rancher-monitoring. #22389

Open bentastic27 opened 5 years ago

bentastic27 commented 5 years ago

What kind of request is this (question/bug/enhancement/feature request): Feature Request

Details:

A customer would like the ability to apply their own prometheus rules to rancher-monitoring. They had attempted to edit the config map only to find that it was overwritten. Currently there is no supported way to add rules and it seems that currently applying them as an answer has risk of breaking the monitoring functionality.

In the regular prometheus chart in the catalog it looks as though they can be added via the serverFiles.rules answer.

Customer's initial email about it:

We are currently rolling out a new deployment of Rancher. In advance of launch, we are setting up alarms and monitoring. However for the webhook to work, we need to make edits to Prometheus.rules and add additional annotations such object, objectClass and so on. In the Rancher UI, we edited the config map Prometheus-cluster-monitoring-rulefiles-0 (specified as the configMap volumemount in statefulset manifest), but the config does not take. We save and then go back to the config and it is still empty.

Addendum:

Will be beneficial to edit the prometheus.yml and alertmanager.yml as well. Customer needs all of these as a package deal to get their alerts to work.

currently our monitoring team make edits to the following files:

  1. prometheus.rules
  2. prometheus.yml
  3. alertmanager.yml

This enables them to

  1. Define rules with custom annotations e.g. object_class which are used by a BPPM webhook
  2. Define jobs and instances , usable in alert definitions for instance
  3. Control alert generation, for example we need the ability to group alerts by alertname and instance, used to prevent alarm storming. This is indispensable.

In rancher, none of this is achievable as configmaps are non-editable/read-only ( The UI never warns about either).

granttr commented 5 years ago

Hi, I was hoping to use Rancher monitoring and need to add some extra prometheus job targets into prometheus.yml - do you know if that works? I have Rancher 2.2.7 and was about to test this... would appreciate any comments if you think this will work or not? Thanks

zhihui921016 commented 4 years ago

I have the same problem. I tried to add a new rulefile in config map Prometheus-cluster-monitoring-rulefiles-0 but it is always restored.

tainguyenbp commented 2 years ago

I have the same problem. I tried to add a new rulefile in config map Prometheus-cluster-monitoring-rulefiles-0 but it is always restored.

hello, do you have custom rule with configmap prometheus