custom alert rules message, rules.annotations.message doesn't get parsed with variable

[teochenglim]

Trying to customise the alert rule message but didn't get the variable parsed.

In config.libsonet

  showMultiCluster: true,
  clusterLabel: 'eks-staging',

In file alerts/apps_alerts.libsonnet

line 5, 6

    prefixedNamespaceSelector: if self.namespaceSelector != null then self.namespaceSelector + ',' else '',
    clusterLabel: self.clusterLabel

line 22

            annotations: {
              message: 'Cluster, %(clusterLabel)s, Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is restarting {{ printf "%.2f" $value }} times / 5 minutes.',
            },

run make

$ make prometheus_alerts.yaml

Result:

      "message": "Cluster: %(clusterLabel), Pod {{ $labels.cluster }}/{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is restarting {{ printf \"%.2f\" $value }} times / 5 minutes."

desired result

      "message": "Cluster: "eks-staging", Pod {{ $labels.cluster }}/{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is restarting {{ printf \"%.2f\" $value }} times / 5 minutes."

[brancz]

clusterLabel is the label-name that describes the cluster, not the cluster name itself. If you set the clusterLabel then you need to make sure it’s part of the alert when it goes out, for example by setting the external labels on your prometheus server.

[teochenglim]

@brancz Thanks for the answer. In my prometheus I have inserted the variable per cluster using the "cluster" keyword. This is inline with Grafana dashboard. Prometheus rules also works prefect. The left over is the alert rule message doesn't tell its origin.

Basically I have federated out the prometheus metrics on each cluster to a centralise prometheus and now I want use this centralise metrics to send alerts.

I wish to find a good solutions that when I increase the number of k8s cluster, this framework shall able to scale with one simple variable changed.

Btw I am using prometheus-operator.

For example

up{cluster="eks-staging"} and up{cluster="eks-production"} is my valid labels

So in other word I can do this right?

"message": "Pod {{ $labels.cluster }}/{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) is restarting {{ printf \"%.2f\" $value }} times / 5 minutes."

[teochenglim]

@brancz btw my actual question is why "%(clusterLabel)s" doesn't get parsed into string. Many thanks.

[cbeneke]

@teochenglim do you use this as formatstring anywhere else? Otherwise it should work fine when you add the key mapping:

annotations: {
  message: 'Cluster %(clusterLabel)s, [...].' % obj,
},

(replace obj with the object holding the clusterLabel object)