[stable/prometheus-operator] how to add additional labels to the default Prometheus alerts for kubernetes system alerts.

aliartiza75 commented 5 years ago

Hello everyone,

I am using prometheus-operator chart for monitoring.

Problem Statement

I want to assign custom labels to default Prometheus alerts?

What I did

I used defautlRules.labels.kind parameter to assign a label to Prometheus alerts but that label was applied to rules metadata level instead of individual rules.

Label value

defaultRules:
labels:
 testLabel: testing

Applied values

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: test-kubernetes-resources
  labels:
    app: test

    chart: prometheus-operator-5.0.13
    release: "RELEASE-NAME"
    heritage: "Tiller"
    expose: "true"

    testLabel: testing

spec:
  groups:
  - name: kubernetes-resources
    rules:
    - alert: KubeCPUOvercommit
      annotations:
        message: Cluster has overcommitted CPU resource requests for Pods and cannot tolerate node failure.
        runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit
      expr: |-
        sum(namespace_name:kube_pod_container_resource_requests_cpu_cores:sum)
          /
        sum(node:node_num_cpu:sum)
          >
        (count(node:node_num_cpu:sum)-1) / count(node:node_num_cpu:sum)
      for: 5m
      labels:
        severity: warning

Instead, I want the label on this alert level like this:


apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: test-kubernetes-resources
labels:
app: test

chart: prometheus-operator-5.0.13
release: "RELEASE-NAME"
heritage: "Tiller"
expose: "true"

spec: groups:

name: kubernetes-resources rules:
- alert: KubeCPUOvercommit annotations: message: Cluster has overcommitted CPU resource requests for Pods and cannot tolerate node failure. runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubecpuovercommit expr: |- sum(namespace_name:kube_pod_container_resource_requests_cpu_cores:sum) / sum(node:node_num_cpu:sum)
  (count(node:node_num_cpu:sum)-1) / count(node:node_num_cpu:sum) for: 5m labels: severity: warning testLabel: testing
```
After looking at the template file I found out that there is no way to assign `custom label` to `alerts`
```

Proposal

I want to change the structure of defaultRules.rules.kubernetesResources from boolean to json with following keys:

defaultRules:
   rules:
       kubernetesResources:
           enabled: boolean
           labels:
               label1: label1Value
               label2: label2Value

enabled key will be used to enable and disable the KubernetesResources alerts.
labels keys values will be used to assign custom labels to alerts labels parameter.

I want to make the above changes for these alerts.

Parameter	Description	Default
`defaultRules.rules.kubeApiserver`	Create Api Server default rules	`true`
`defaultRules.rules.kubePrometheusNodeAlerting`	Create Node Alerting default rules	`true`
`defaultRules.rules.kubePrometheusNodeRecording`	Create Node Recording default rules	`true`
`defaultRules.rules.kubeScheduler`	Create Kubernetes Scheduler default rules	`true`
`defaultRules.rules.kubernetesAbsent`	Create Kubernetes Absent (example API Server down) default rules	`true`
`defaultRules.rules.kubernetesApps`	Create Kubernetes Apps default rules	`true`
`defaultRules.rules.kubernetesResources`	Create Kubernetes Resources default rules	`true`
`defaultRules.rules.kubernetesStorage`	Create Kubernetes Storage default rules	`true`
`defaultRules.rules.kubernetesSystem`	Create Kubernetes System default rules	`true`

Kindly reply asap as this change is required for a task I am working on.

Thank you

aliartiza75 commented 5 years ago

Guys @davidkarlsen, @kflynn, kindly have a look at this issue. Thank you

aliartiza75 commented 5 years ago

@carrodher @javsalgar @AdamDang guys kindly have a look at the above issue.

Thank you

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

aliartiza75 commented 5 years ago

@carrodher @javsalgar @AdamDang kindly have a look at this issue.

carrodher commented 5 years ago

Ey @aliartiza75, I'm sorry but I don't have experience using this chart. According to https://github.com/helm/charts/blob/master/stable/prometheus-operator/OWNERS try to ping @gianrubio @vsliouniaev @anothertobi. Surely they can help you regarding this issue

aliartiza75 commented 5 years ago

@gianrubio @vsliouniaev @anothertobi kindly have a look at this issue.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

vsliouniaev commented 5 years ago

@aliartiza75 that isn't possible to achieve with the chart. I suggest that any modifications to the alerts that you wish to do should be done yourself, deploying the custom alerts separately to the chart. The alerts are all sourced from several upstreams, which means that the chart would have to add custom templating logic to expose this kind of functionality.

aliartiza75 commented 5 years ago

@vsliouniaev thank you for response.

djsly commented 4 years ago

We have the same requirements.. We need to be able to route the alerts based on labels...

P6rguVyrst commented 4 years ago

Added an example here recently: "Customising alert annotations"

The alerts in the chart are copied from: https://github.com/coreos/kube-prometheus/tree/master/jsonnet/kube-prometheus The example in kubernetes-monitoring can also be applied to the coreos/kube-prometheus context if needed.

It's a waste of effort to copy and maintain these alerts in a helm chart.

More info on the concept: https://monitoring.mixins.dev/

bartoszcisek commented 3 years ago

Custom rule labels can be added now via regular helm values using variable additionalRuleLabels

helm / charts