prometheus / alertmanager

Prometheus Alertmanager
https://prometheus.io
Apache License 2.0
6.43k stars 2.12k forks source link

[Feature] Support adaptive cards for MSTeams #3503

Open aw1cks opened 10 months ago

aw1cks commented 10 months ago

Hi there,

I saw that MS Teams support was recently added, many thanks for that! I've been using prometheus-msteams at work for quite some time and it will be great to have the integration natively.

It would be awesome if support for the newer adaptive cards message format could be added at some point.

Here's the payload needed to fire these into Teams:

{
   "type":"message",
   "attachments":[
      {
         "contentType":"application/vnd.microsoft.card.adaptive",
         "contentUrl":null,
         "content":{
           <user provided JSON for adaptive card goes here>
         }
      }
   ]
}

And here's a sample alert that I created: https://gist.github.com/aw1cks/20c60986e789342a5e1e847b5d05a954

Proposal

Add a configuration option for the msteams integration, to switch the payload format to adaptive web cards, delegating responsibility for creating valid adaptive card JSON to the user's template.

One complication is that it's not trivial to keep a separate title and text template per current implementation - but I'd propose that if the aforementioned config option were enabled, that it'd be left to the user to template the title as they saw fit, and just read the text field, placing it directly into the content field of the above JSON payload.

If such an approach would be accepted, I'd be happy to look at doing this!

aw1cks commented 10 months ago

I did play around with this - here's a somewhat hacky, but working change: https://github.com/aw1cks-forks/alertmanager/commit/5fca6eb4f68b516d52b13975e914ce12c08ae48d I'd imagine it'd be best to keep the existing MessageCard implementation and allow users to opt into adaptive cards if desired.

grobinson-grafana commented 10 months ago

I added support for Adaptive Cards to Grafana. Perhaps we can use the same code for Alertmanager too? @gotjosh

aw1cks commented 10 months ago

Nice! That seems like a path of low resistance.

My only comment is that the card format appears to be quite prescriptive, which does limit the cusomisability.

6fears7 commented 3 months ago

@aw1cks Been playing around with this today. Got some nice dynamic creation of the Cards to work well all through just passing in the JSON data inside a .tmpl file that was defining the text. Awesome stuff.

Can you consider opening a PR for your changes? It makes the alerting experience in Teams much better.

Firing

image

Resolved

image

Label / Annotations View:

image

Silences:

image

{{ define "new.text" }}
{
    "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
    "type": "AdaptiveCard",
    "version": "1.2",
    "padding": "None",
    "msteams": {
        "width": "Full"
    },

    "body": [
        {
            "type": "Container",
            "id": "alert-msg",
            "padding": "Default",
            "items": [
                {
                    "type": "TextBlock",
                    "id": "alert-summary-title",
                    "text": "[{{- if gt (len .Alerts.Firing) 0 }}FIRING: {{ .Alerts.Firing | len }}] 🔥 {{end}} {{- if gt (len .Alerts.Resolved) 0 }}RESOLVED: {{ .Alerts.Resolved | len }}] ✅ {{end}} {{ with index .Alerts 0 -}}{{ .Labels.alertname }}{{ end }}",
                    "weight": "Bolder",
                    "color": "{{- if gt (len .Alerts.Firing) 0 }}Attention{{end}}{{- if gt (len .Alerts.Resolved) 0 }}Good{{end}}",
                    "size": "ExtraLarge",
                    "horizontalAlignment": "Left"
                },
                {
                    "type": "Container",
                    "id": "alert-summary-container",
                    "padding": "None",
                    "items": [
                        {
                            "type": "TextBlock",
                            "id": "alert-summary-description",
                            "text": "{{ .Alerts.Firing | len }} alerts are firing",
                            "wrap": true
                        },
                         {
                            "type": "ActionSet",
                            "id": "alert-silence-action",
                            "actions": [
                                {
                                    "type": "Action.OpenUrl",
                                    "id": "silence",
                                    "title": "Silence",
                                    "url": "{{ .ExternalURL }}/#/silences/new?filter=%7B
                                    {{- range $key, $value := .CommonLabels }}
                                    {{- if eq $key "alertname" }}{{ $key }}%3D%22{{ reReplaceAll "\\\\" "" $value }}%22{{ end }}
                                    {{- end }}
                                    {{- range $key, $value := .CommonLabels }}
                                    {{- if ne $key "alertname" }}%2C{{ $key }}%3D%22{{ reReplaceAll "\\\\" "" $value }}%22{{ end }}
                                    {{- end -}}%7D"
                                },
                                {{ with $alert := index .Alerts 0}}
                                {
                                    "type": "Action.OpenUrl",
                                    "id": "prom",
                                    "title": "View in Prometheus",
                                    "url": "{{ .GeneratorURL }}"
                                }

                                {{ end }}      

                            ]
                        }

                    ]
                },
                 {{ range $index, $alert := .Alerts.Firing }}
                {
                    "type": "Container",
                    "id": "{{ $index }}-alerts-container",
                    "isVisible": true,
                    "padding": "None",
                    "items": [

                        {
                            "type": "Container",
                            "id": "alert-{{ $index }}-msg-container",
                            "padding": "None",
                            "separator": true,
                            "items": [
                                {
                                    "type": "TextBlock",
                                    "id": "alert-{{ $index }}-summary",
                                    "text": "{{ $alert.Labels.alertname}}",
                                    "wrap": true,
                                    "size": "Medium",
                                    "weight": "Bolder"
                                },
                                {
                                    "type": "TextBlock",
                                    "id": "alert-{{ $index }}-description",
                                    "text": "{{ $alert.Annotations.description }}",
                                    "wrap": true,
                                    "weight": "Lighter",
                                    "size": "Small"
                                },
                                {
                                    "type": "ActionSet",
                                    "id": "alert-{{ $index }}-actions",
                                    "actions": [
                                        {
                                            "type": "Action.ShowCard",
                                            "title": "View Labels",
                                            "card": {
                                                "type": "AdaptiveCard",
                                                "body": [
                                                    {
                                                    "type": "TextBlock",
                                                    "wrap": true,
                                                    "text": "{{ $alert.Labels }}"
                                                    }
                                                ]
                                            }
                                        },
                                          {
                                            "type": "Action.ShowCard",
                                            "title": "View Annotations",
                                            "card": {
                                                "type": "AdaptiveCard",
                                                "body": [
                                                    {
                                                    "type": "TextBlock",
                                                    "wrap": true,
                                                    "text": "{{ $alert.Annotations }}"
                                                    }
                                                ]
                                            }
                                        }

                                    ]
                                },

                                {
                                    "type": "Container",
                                    "id": "alert-{{ $index }}-backup-labels-container",
                                    "padding": "None",
                                    "isVisible": false,
                                    "items": [
                                        {
                                            "type": "FactSet",
                                            "id": "alert-{{ $index }}-backup-labels-factset",
                                            "facts": []
                                        }

                                    ]
                                }
                            ]
                        }
                    ]
                },
                {{ end }}
                {{ range $index, $alert := .Alerts.Resolved }}
                {
                    "type": "Container",
                    "id": "{{ $index }}-alerts-container",
                    "isVisible": true,
                    "padding": "None",
                    "items": [

                        {
                            "type": "Container",
                            "id": "alert-{{ $index }}-msg-container",
                            "padding": "None",
                            "separator": true,
                            "items": [
                                {
                                    "type": "TextBlock",
                                    "id": "alert-{{ $index }}-summary",
                                    "text": "{{ $alert.Labels.alertname}}",
                                    "wrap": true,
                                    "size": "Medium",
                                    "weight": "Bolder"
                                },
                                {
                                    "type": "TextBlock",
                                    "id": "alert-{{ $index }}-description",
                                    "text": "{{ $alert.Annotations.description }}",
                                    "wrap": true,
                                    "weight": "Lighter",
                                    "size": "Small"
                                },
                                {
                                    "type": "ActionSet",
                                    "id": "alert-{{ $index }}-actions",
                                    "actions": [
                                        {
                                            "type": "Action.ShowCard",
                                            "title": "View Labels",
                                            "card": {
                                                "type": "AdaptiveCard",
                                                "body": [
                                                    {
                                                    "type": "TextBlock",
                                                    "wrap": true,
                                                    "text": "{{ $alert.Labels }}"
                                                    }
                                                ]
                                            }
                                        },
                                          {
                                            "type": "Action.ShowCard",
                                            "title": "View Annotations",
                                            "card": {
                                                "type": "AdaptiveCard",
                                                "body": [
                                                    {
                                                    "type": "TextBlock",
                                                    "wrap": true,
                                                    "text": "{{ $alert.Annotations }}"
                                                    }
                                                ]
                                            }
                                        }

                                    ]
                                },

                                {
                                    "type": "Container",
                                    "id": "alert-{{ $index }}-backup-labels-container",
                                    "padding": "None",
                                    "isVisible": false,
                                    "items": [
                                        {
                                            "type": "FactSet",
                                            "id": "alert-{{ $index }}-backup-labels-factset",
                                            "facts": []
                                        }

                                    ]
                                }
                            ]
                        }
                    ]
                },
                {{ end }}
                {
                    "type": "Container",
                    "id": "backup-alerts-container",
                    "isVisible": false,
                    "padding": "None",
                    "items": [
                        {
                            "type": "Container",
                            "id": "alert-catch-msg-container",
                            "padding": "None",
                            "separator": false,
                            "items": []
                        }
                    ]
                }
            ]
        }

    ]
}
{{ end }}
receivers:
  - name: sandbox
    msteams_configs:
      -  webhook_url: "YOUR_WEBHOOK_HERE"
         text: '{{ template "new.text" . }}'

Then I just send a POST with my JSON alert data to the Alertmanager /api/v2/alerts endpoint.

mac2000 commented 1 month ago

To me it seems like this one is not something that may fit everyones needs and indeed forming json via go templating is a portal to hell

For anyone looking for workaround you may do very simple trick:

From alertmanager side configure webhook receiver, aka:

receivers:
  - name: default
    webhook_configs:
      - send_resolved: false
        url: http://localhost:8080/demo

alertmanager will just send raw json payload to given url

and now you can code some simple, single endpoint service, that will take given input, transform it to whatever you wish to have in teams and send it to teams, aka:

image

Use designer to form message and samples for inspiration

Here is an example ```js const body = { receiver: 'default', status: 'firing', alerts: [ { status: 'firing', labels: { alertname: 'demo2', component: 'bar', severity: 'info', }, annotations: {}, startsAt: '2024-05-13T06:11:03.188793747Z', endsAt: '0001-01-01T00:00:00Z', generatorURL: '', fingerprint: 'c8002adfd87cf31c', }, { status: 'firing', labels: { alertname: 'demo2', component: 'foo', severity: 'info', }, annotations: {}, startsAt: '2024-05-13T06:11:03.177669584Z', endsAt: '0001-01-01T00:00:00Z', generatorURL: '', fingerprint: 'd31cdf353f33ac5b', }, ], groupLabels: { alertname: 'demo2', severity: 'info' }, commonLabels: { alertname: 'demo2', severity: 'info' }, commonAnnotations: {}, externalURL: 'http://eb024ed65dcf:9093', version: '4', groupKey: '{}:{alertname="demo2", severity="info"}', truncatedAlerts: 0, } const email = body?.groupLabels?.annotation_owner || body?.commonLabels?.annotation_owner || body?.groupLabels?.owner || body?.commonLabels?.owner || body?.groupLabels?.tag_owner || body?.commonLabels?.tag_owner || body?.alerts?.[0]?.labels?.annotation_owner || body?.alerts?.[0]?.labels?.owner || body?.alerts?.[0]?.labels?.tag_owner || 'mac@mac-blog.org.ua' // '' const payload = { type: 'message', attachments: [ { contentType: 'application/vnd.microsoft.card.adaptive', content: { $schema: 'http://adaptivecards.io/schemas/adaptive-card.json', version: '1.0', type: 'AdaptiveCard', body: [ { type: 'TextBlock', weight: 'default', text: `${email}`, }, ], msteams: { entities: [ { type: 'mention', text: `${email}`, mentioned: { id: email, name: email, }, }, ], }, }, }, ], } for (const alert of body.alerts) { const table = { type: 'Table', columns: [{ width: 1 }, { width: 1 }], rows: [], } for (const [key, val] of Object.entries(alert.labels)) { table.rows.push({ type: 'TableRow', cells: [ { type: 'TableCell', items: [ { type: 'TextBlock', text: key, wrap: true, weight: 'default', }, ], }, { type: 'TableCell', items: [ { type: 'TextBlock', text: val, wrap: true, weight: key === 'alertname' ? 'bolder' : 'default', // 'bolder' color: 'default', // 'attention', 'good', 'warning' depending on key and val, aka status=firing - attention, severity=warning - warning }, ], }, ], }) } payload.attachments[0].content.body.push(table) const actions = { type: 'ActionSet', actions: [], } if (alert.generatorURL) { actions.actions.push({ type: 'Action.OpenUrl', title: 'prometheus', url: alert.generatorURL, }) } if (body.externalURL) { actions.actions.push({ type: 'Action.OpenUrl', title: 'alertmanager', url: body.externalURL, }) actions.actions.push({ type: 'Action.OpenUrl', title: 'silence', url: body.externalURL, // TODO: build silence link }) } actions.actions.push({ type: 'Action.OpenUrl', title: 'readme', url: `https://mac.atlassian.net/wiki/search?spaces=OPS&text=${alert.labels.alertname}`, }) payload.attachments[0].content.body.push(actions) } console.log(JSON.stringify(payload, null, 4)) const res = await fetch( 'https://mac.webhook.office.com/webhookb2/.../IncomingWebhook/...', { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify(payload), } ) console.log(res.status) // 200 console.log(res.statusText) // 'OK' console.log(await res.text()) // '1' ``` > Note: js is used here only for example, if you wish you may go get alertmanager models directly, or even write everything as lua script in nginx