[kube-prometheus-stack] msteams_configs not found in type config.plain

[SergkeiM]

Describe the bug a clear and concise description of what the bug is.

After upgrading to 48.4.0 and trying to use msteams in alertmanager:

receivers:
      - name: 'msteams'
        msteams_configs:
          - webhook_url: '<url>'
            send_resolved: true
            title: '{{ if eq .Status "firing" }}🚨 **FIRING** 🔥{{- else -}}🙌 **RESOLVED** 🍻{{- end -}}'
            text: '{{ template "msteams.text" . }}'

I get this error:

sync "kube-prometheus-stack/kube-prometheus-stack-alertmanager" failed: provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors: line 32: field msteams_configs not found in type config.plain

What's your helm version?

v3.10.3

What's your kubectl version?

v1.23.6

Which chart?

kube-prometheus-stack

What's the chart version?

48.4.0

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm upgrade kube-prometheus-stack prometheus-community/kube-prometheus-stack \ --version 48.4.0 \ --namespace kube-prometheus-stack \ --values values.yml

Anything else we need to know?

No response

[arukiidou]

@SergkeiM

• prometheus-operator supports v0.25.0 yet.
  • Please wait for crd updates to support additional configurations.
  • upstream upgrade is coming prometheus-operator/kube-prometheus#2207
  • next release cut (v68.0.0)

[dbluxo]

Hey @SergkeiM 👋 , just out of curiosity and for inspiration, could you please share your "msteams.text" template?

[SergkeiM]

Hi @dbluxo

I have 2 templates for msteams this one I use in ceph alertmanager (HTML)

title: '{% raw %}{{ if eq .Status "firing" }}🚨 <b>FIRING</b> 🔥{{- else -}}🙌 <b>RESOLVED</b> 🍻{{- end -}}{% endraw %}'
text: '{% raw %}
<b>Alert:</b> <code>{{ .CommonLabels.alertname }}</code><br>
<b>Severity:</b> <code>{{ .CommonLabels.severity | title -}}</code><br>
{{- if (index .Alerts 0).Annotations.summary }}<b>Summary:</b> {{ (index .Alerts 0).Annotations.summary }}<br><br>{{- end }}
{{ range .Alerts }}
{{- if .Annotations.description }}{{ .Annotations.description }}<br><br>{{- end }}
{{- if .Annotations.message }}{{ .Annotations.message }}<br><br>{{- end }}
{{- end }}{% endraw %}'

And this one in Grafana and Prometheus

title: '{{ if eq .Status "firing" }}🚨 **FIRING** 🔥{{- else -}}🙌 **RESOLVED** 🍻{{- end -}}'
text: '{{ template "msteams.text" . }}'

msteams.tmpl: |
      {{ define "msteams.text" -}}
      **Alert:** {{ .CommonLabels.alertname }}\
      **Service:** {{ .CommonLabels.service }}\
      **Severity:** {{ .CommonLabels.severity | title -}}\
      {{- if (index .Alerts 0).Annotations.summary }}
      **Summary:** {{ (index .Alerts 0).Annotations.summary }}
      {{- end }}
      {{ range .Alerts }}
      {{- if .Annotations.description }}{{ .Annotations.description }}\{{- end }}
      {{- if .Annotations.message }}{{ .Annotations.message }}\{{- end }}
      {{- end }}
      {{- end }}

For some reason HTML doesn't work in Prometheus (Alertmanager) so had to use markdown.

[EraYaN]

This is still a problem with v49 of the chart (with updated alertmanager etc).

[SergkeiM]

Hi @EraYaN

This is becuase prometheus is not yet updated the image: https://quay.io/repository/prometheus/prometheus?tab=tags&tag=latest

The last version kube-prometheus-stack uses is: 'v2.46.0', but msteams was added after, it is available in main though

https://github.com/prometheus-community/helm-charts/blob/297128656e600fa2cbe08881d5945a51326d653a/charts/kube-prometheus-stack/values.yaml#L2897

So we need to wait until 'prometheus' image is released (v2.47.0), and after chart is updated.

But is defintley confusing, I think first all related things has to be updated, and after the charts, not vice-versa.

[SergkeiM]

Hi @arukiidou the above was fixed with release of 51.0.0, but there is a new error now:

failed: provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors:\n line 32: field msteams_configs not found in type alertmanager.receiver

[zeritti]

Support for alertmanager's msteams_configs did not make it to the latest release of Prometheus operator (0.68.0). The feature cannot be used with the operator as yet (is being worked on in PR 5778).

[arukiidou]

To make matters worse, I worked on the prometheus bump alertmanager, but did not make it in v2.47.0. It will v2.48.0.

[tadavid-cae]

and now what?

[zeritti]

Support for the msteams receiver is present in the latest Prometheus operator release 0.69.0. You can try that out (with CRDs from the same release) or wait until the next chart's major release is out.

[95gabor]

With @tadavid-cae we found that the issue was fixed in: https://github.com/prometheus/alertmanager/pull/3495, and it has not been released yet, but it is merged to the main branch. To use the fixed version, you need to override the alert manager tag in your helm values files: (until a new version is released)

alertmanager:
  tag: main@sha256:1c3941499c0379e516b75fcfa28f991342e12e922cdf499620a7e4b33dd5060e

[SergkeiM]

Hi @95gabor I don't think is alertmanager issue, becuase I do use MSTeams in ceph with alertmanager 0.26, also works in grafana, the only problems is prometheus and we wait for 2.48

[tadavid-cae]

@SergkeiM Probably I am missing something, but how would prometheus be problematic here when sending alerts to teams is not working which is done by alertmanager?

[bmgante]

Can anyone confirm if latest version on kube-prometheus-stack already addresses this problem?

[mihaico]

I can confirm this is working with kube-prometheus-stack:54.1.0, the only comment is that one needs to specify both text and title due to a bug that was fixed post-release in prometheus-operator https://github.com/prometheus-operator/prometheus-operator/pull/6109

here is an example:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: test-service
  namespace: test-service
  labels:
    app: alertmanager-config
spec:
  route:
    receiver: 'test-service'
    routes:
    - receiver: 'test-service'
      continue: true
      matchers:
      - name: 'namespace'
        matchType: '='
        value: 'test-service'
      - name: 'severity'
        matchType: '=~'
        value: 'warning|critical|WARNING|CRITICAL'
  receivers:
  - name: 'test-service'
    msteamsConfigs:
    - webhookUrl:
        key: webhook_url
        name: test-service-secret
      sendResolved: true
      title: '{{ template "msteams.default.title" . }}'
      text: '{{ template "msteams.default.text" . }}'

[bmgante]

@mihaico lastest chart version is 54.2.2, does that version still requires that workaround?

[mihaico]

@mihaico lastest chart version is 54.2.2, does that version still requires that workaround?

I see its using prometheus-operator:0.69.1 release, so I believe it still does need the workaround.

[bmgante]

Ok thanks for confirming @mihaico . I'll give it a try because I am currently using another deployment (prometheus-msteams) to be able to send notifications to teams via alertmanager and this default integration is way better 😉

[SergkeiM]

Confirm, in Helm chart version 54.2.2 works fine.

[trallnag]

Weird, not working for me yet.

• Chart version 55.0.0
• Prom operator version 0.70.0
• Alertmanager verion 0.26.0

level=error ts=2023-12-05T07:52:38.675552699Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="sync \"observability/kps-kube-prometheus-stack-alertmanager\" failed: provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors:\n line 26: field msteams_config not found in type config.plain\n line 32: field msteams_config not found in type config.plain\n line 38: field msteams_config not found in type config.plain\n line 44: field msteams_config not found in type config.plain\n line 50: field msteams_config not found in type config.plain\n line 56: field msteams_config not found in type config.plain\n line 62: field msteams_config not found in type config.plain\n line 68: field msteams_config not found in type config.plain\n line 74: field msteams_config not found in type config.plain\n line 80: field msteams_config not found in type config.plain"

[SergkeiM]

Hi @trallnag

They key is: msteams_configs not msteams_config

example:

receivers:
      - name: 'msteams'
        msteams_configs:
          - webhook_url: 'WEBHOOK_URLf'
            send_resolved: true
            title: '{{ if eq .Status "firing" }}🚨 [FIRING] 🔥{{- else -}}🙌 [RESOLVED] 🍻{{- end -}}'
            text: '{{ template "msteams.text" . }}'

[trallnag]

Amazing, nice catch. Thanks!

[borsenkOoleh]

Hi @dbluxo

I have 2 templates for msteams this one I use in ceph alertmanager (HTML)

title: '{% raw %}{{ if eq .Status "firing" }}🚨 <b>FIRING</b> 🔥{{- else -}}🙌 <b>RESOLVED</b> 🍻{{- end -}}{% endraw %}'
text: '{% raw %}
<b>Alert:</b> <code>{{ .CommonLabels.alertname }}</code><br>
<b>Severity:</b> <code>{{ .CommonLabels.severity | title -}}</code><br>
{{- if (index .Alerts 0).Annotations.summary }}<b>Summary:</b> {{ (index .Alerts 0).Annotations.summary }}<br><br>{{- end }}
{{ range .Alerts }}
{{- if .Annotations.description }}{{ .Annotations.description }}<br><br>{{- end }}
{{- if .Annotations.message }}{{ .Annotations.message }}<br><br>{{- end }}
{{- end }}{% endraw %}'

And this one in Grafana and Prometheus

title: '{{ if eq .Status "firing" }}🚨 **FIRING** 🔥{{- else -}}🙌 **RESOLVED** 🍻{{- end -}}'
text: '{{ template "msteams.text" . }}'

msteams.tmpl: |
      {{ define "msteams.text" -}}
      **Alert:** {{ .CommonLabels.alertname }}\
      **Service:** {{ .CommonLabels.service }}\
      **Severity:** {{ .CommonLabels.severity | title -}}\
      {{- if (index .Alerts 0).Annotations.summary }}
      **Summary:** {{ (index .Alerts 0).Annotations.summary }}
      {{- end }}
      {{ range .Alerts }}
      {{- if .Annotations.description }}{{ .Annotations.description }}\{{- end }}
      {{- if .Annotations.message }}{{ .Annotations.message }}\{{- end }}
      {{- end }}
      {{- end }}

For some reason HTML doesn't work in Prometheus (Alertmanager) so had to use markdown.

Hi @SergkeiM Unfortunately, I can't find any documentation on how to add the custom template to kube-prometheus-stack values.yaml Please explain to me how to do it. I have tried to do it with alertmanagerFiles, but it was not working