Feature request: fill placeholders in alert.

[andrewhowdencom]

Current getting messages which are like $lablels.instance is down which is somewhat disconcerting ;)

Happy to look at a PR at some point, but my golang is awful.

[Swatto]

I understand your point but the code here just read the received JSON from Prometheus (you should be in total control of that). I don't want to change the text sended since it's not the responsability of this program: KISS principle.

Feel free to argue of course.

[andrewhowdencom]

I think the placeholders are sent as part of that JSON payload, in the lables{} object. See:

• https://prometheus.io/docs/alerting/configuration/#webhook-receiver-webhook_config

Grafana does some replacing of the alerts I gather; or, it's querying in such as way they're already populated.

At any rate, I'll take a look further at this and get back to you.

Also, props for the super fast feedback!

[Swatto]

Thanks for your explanation. Feel free to send me the content of the annotations object of the alerts array inside the JSON sended by the alert manager (I don't have time to check it by myself).

With that, we could continue to discuss and look at what we could do.

[andrewhowdencom]

Stolen out of tcpdump

{
  "receiver": "default-receiver",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "InstanceDown",
        "instance": "http://pgp.andrewhowden.com",
        "job": "blackbox"
      },
      "annotations": {
        "description": "Unable to scrape $labels.instance",
        "summary": "Address $labels.instance appears to be down"
      },
      "startsAt": "2017-01-06T19:34:52.887Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus-prometheus-3976174244-lgcwn:9090/graph?g0.expr=probe_success%7Bjob%3D%22blackbox%22%7D+%3D%3D+0&g0.tab=0"
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "InstanceDown",
        "instance": "http://www.andrewhowden.com",
        "job": "blackbox"
      },
      "annotations": {
        "description": "Unable to scrape $labels.instance",
        "summary": "Address $labels.instance appears to be down"
      },
      "startsAt": "2017-01-06T19:34:52.887Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus-prometheus-3976174244-lgcwn:9090/graph?g0.expr=probe_success%7Bjob%3D%22blackbox%22%7D+%3D%3D+0&g0.tab=0"
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "InstanceDown",
        "instance": "http://andrewhowden.com",
        "job": "blackbox"
      },
      "annotations": {
        "description": "Unable to scrape $labels.instance",
        "summary": "Address $labels.instance appears to be down"
      },
      "startsAt": "2017-01-06T19:34:52.887Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus-prometheus-3976174244-lgcwn:9090/graph?g0.expr=probe_success%7Bjob%3D%22blackbox%22%7D+%3D%3D+0&g0.tab=0"
    }
  ],
  "groupLabels": {
    "alertname": "InstanceDown"
  },
  "commonLabels": {
    "alertname": "InstanceDown",
    "job": "blackbox"
  },
  "commonAnnotations": {
    "description": "Unable to scrape $labels.instance",
    "summary": "Address $labels.instance appears to be down"
  },
  "externalURL": "http://prometheus-prometheus-3976174244-lgcwn:9093",
  "version": "3",
  "groupKey": 9777663806026785000
}

Alert configuration

https://github.com/littlemanco/k8s.instrumentation/blob/master/deployment/charts/prometheus/alerts#L13-L19

Or,

# Alert is any instance goes down

ALERT InstanceDown
  IF probe_success{job="blackbox"} == 0
  FOR 5m
  ANNOTATIONS {
    summary = "Address $labels.instance appears to be down",
    description = "Unable to scrape $labels.instance"
  }

[Swatto]

Ok now I completely understand your needs. I will tackle that at the beginning of the week.

By the way, can I use those data for test writing if I remove your DNS reference ?

[andrewhowdencom]

Go nuts. It's all information I'm prepared to have public anyway -- I use it as a teaching tool.

Thanks for considering the feature (: I appreciate it! Pretty excited about wiring together Prometheus in a meaningful way.

[Swatto]

The feature is implemented here c1153bce12b1395fd5cdd17046db44950af8def3. You can find a new version of the Docker image online (tag is :latest or :1.1).

[andrewhowdencom]

<3 bro you are amazing! I will test it now.

[Swatto]

Sure and tell me if it's ok.