Error sending event to webhook Summary or Text is required for MSTeams

[ramakrishnateja]

Describe the bug

Here is my canary definition

apiVersion: flagger.app/v1beta1 kind: Canary metadata: name: helloweather spec:

service mesh provider can be: kubernetes, istio, appmesh, nginx, gloo

provider: kubernetes

deployment reference

targetRef: apiVersion: apps/v1 kind: Deployment name: helloweather-deployment

the maximum time in seconds for the canary deployment

to make progress before rollback (default 600s)

progressDeadlineSeconds: 60

HPA reference (optional)

autoscalerRef: apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler name: helloweather-hpa service: name: helloweather port: 80 portDiscovery: true analysis:

schedule interval (default 60s)

interval: 30s
# max number of failed checks before rollback
threshold: 2
# number of checks to run before rollback
iterations: 10
# Prometheus checks based on 
# http_request_duration_seconds histogram
metrics:
  - name: request-success-rate
    # minimum req success rate (non 5xx responses)
    # percentage (0-100)
    thresholdRange:
      min: 99
    interval: 1m
  - name: request-duration
    # maximum req duration P99
    # milliseconds
    thresholdRange:
      max: 500
    interval: 30s
# acceptance/load testing hooks
webhooks:
  - name: "send to Teams"
    type: event
    url: "https://teamswebhook"
  - name: smoke-test
    type: pre-rollout
    url: http://flagger-loadtester.linkerd/
    timeout: 15s
    metadata:
      type: bash
      cmd: "curl http://helloweather.manualbluegreen:80/WeatherForecast"
  - name: gate
    type: confirm-rollout
    url: http://flagger-loadtester.linkerd/gate/check
  - name: load-test
    url: http://flagger-loadtester.linkerd/
    timeout: 5s
    metadata:
      type: cmd
      cmd: "hey -z 1m -q 10 -c 2 http://helloweather.manualbluegreen:80/WeatherForecast/"
alerts:
  - name: "Error MS Teams"
    severity: error
    providerRef:
      name: "msteams"
  - name: "Warn MS Teams"
    severity: warn
    providerRef:
      name: "msteams"
  - name: "Info MS Teams"
    severity: info
    providerRef:
      name: "msteams"

To Reproduce

When the canary is applied and image for deployment changed i am able to see the events raised by canary by doing a kubectl get events but not seeing the events sent to the MSTeams channel. Its failing with following error.

{"level":"error","ts":"2022-03-14T18:02:17.239Z","caller":"controller/events.go:69","msg":"error sending event to webhook: Summary or Text is required.","canary":"helloweather.manualbluegreen","stacktrace":"github.com/fluxcd/flagger/pkg/controller.(Controller).sendEventToWebhook\n\t/workspace/pkg/controller/events.go:69\ngithub.com/fluxcd/flagger/pkg/controller.(Controller).recordEventInfof\n\t/workspace/pkg/controller/events.go:35\ngithub.com/fluxcd/flagger/pkg/controller.(Controller).runBlueGreen\n\t/workspace/pkg/controller/scheduler.go:620\ngithub.com/fluxcd/flagger/pkg/controller.(Controller).advanceCanary\n\t/workspace/pkg/controller/scheduler.go:410\ngithub.com/fluxcd/flagger/pkg/controller.CanaryJob.Start.func1\n\t/workspace/pkg/controller/job.go:39"}

Expected behavior

Should see the events sent to MSTeams

Additional context

• Flagger version: 1.19.0
• Kubernetes version: 1.21
• Service Mesh provider: linkerd
• Ingress provider: nginx

[aryan9600]

This is the payload that Flagger sends to webhooks:

{
    "name": "podinfo",
    "namespace": "test",
    "phase": "Progressing", 
    "metadata": {
        "test":  "all",
        "token":  "16688eb5e9f289f1991c"
    }
}

This is not compatible with the webhook payload that MS teams requires.

[stefanprodan]

We do have support for MS Teams by setting type: msteams in the AlertProvider, docs here: https://docs.flagger.app/usage/alerting#canary-configuration

[claudiu-albu]

@stefanprodan

We do have support for MS Teams by setting type: msteams in the AlertProvider, docs here: https://docs.flagger.app/usage/alerting#canary-configuration

I'm trying exactly this with following setup:

Flagger AlertProvider:

apiVersion: flagger.app/v1beta1
kind: AlertProvider
metadata:
  name: msteams-notifs
  namespace: flagger
spec:
  type: msteams
  secretRef:
    name: msteams-webhook

Flagger HelmRelease:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: flagger
  namespace: flagger
spec:
  interval: 5m
  chart:
    spec:
      chart: flagger
      version: 1.38.0
      sourceRef:
        kind: HelmRepository
        name: flagger
        namespace: flagger
      interval: 1m
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      remediateLastFailure: false
  values:
    meshProvider: istio
    metricsServer: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090
    clusterName: "${project}-${environment_name}"

and added this bit to Flagger Canary, in spec.analysis:

    alerts:
      - name: "MS Teams Notifications"
        severity: info
        providerRef:
          name: msteams-notifs
          namespace: flagger

I may not be entirely clear about what documentation explains, but I can't for the life of me make it send MS Teams notifications other than:

and: (these might as well be generated by underlying FluxCD alerting given their title "FluxCD Alerts". not entirely sure at the moment.)

this is a typical output for my canary deployment testing:

but even when I make Canary fail altogether, I still don't get anything corresponding in MS Teams, like documentation shows here:

while using setup shown above, I don't get any errors in Flagger k8s pod either while Canary deployment is running.

in Flagger HelmRelease, in spec.values, I've also tried using:

    env:
    - name: MSTEAMS_URL
      valueFrom:
       secretKeyRef:
         name: msteams-webhook
         key: address

or:

    msteams:
      url: <MSTEAMS_WEBHOOK_URL>

but this only results in making error "Error sending event to webhook Summary or Text is required for MSTeams" show up in Flagger pod logs again.

Additional context

• Flagger version: 1.38.0
• Kubernetes version: 1.28.11
• Service Mesh provider: Istio

so am I getting it wrong in terms of what Flagger MS Teams alerting does? what can I actually make Flagger alert on in MS Teams notifications?

any input would be greatly appreciated. thanks in advance, Claudiu