Webhook notifications sometimes sent with GET instead of POST despite configuration

[mgarstecki]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

We configured a webhook notification with a POST, but ArgoCD often calls it with a GET instead. This seems random across many of our apps, but not that rare (several hundred failures per day over a few thousand calls):

Failed to notify recipient {datadog-events } defined in resource argocd/ingress-preprod-ap-aws-1: request to { GET https://api.datadoghq.com/api/v1/events datadog-events} has failed with error code 403 : {"errors":["Forbidden"]}

We've only configured the webhook to be called with a POST, never with a GET.

To Reproduce

This is the relevant notification configuration:

      service.webhook.datadog-events: |
        url: https://api.datadoghq.com/api/v1/events
        headers: #optional headers
        - name: "Accept"
          value: "application/json"
        - name: "Content-Type"
          value: "application/json"
        - name: "DD-API-KEY"
          value: $datadog-api-key
...
    template.datadog-events-release: |
      webhook:
        datadog-events:
          method: POST
          body: |
            {
              "source_type_name": "argocd",
              "tags": [
                "continent:{{ index .app.metadata.labels "continent" }}",
                "env:{{ index .app.metadata.labels "env" }}",
                "cluster:{{ .app.spec.destination.name }}",
                "service:{{ .app.metadata.name }}",
                "source:argocd"
              ],
              "title": "{{ .app.metadata.name }} deployed with commit [{{ .app.status.sync.revision }}] on {{ .app.spec.source.repoURL }}"
            }
...
    trigger.on-deployed: |
      - oncePer: app.status.operationState.syncResult.revision
        send:
        - datadog-events-release
        when: |
          app.status.operationState != nil and app.status.operationState.phase == "Succeeded"
          and
          app.status.health.status == "Healthy"

The template.datadog-events-release section is the only place we call the webhook (so with a POST only).

We don't know what are the exact conditions to reproduce it: some calls do get through (we see the results of the POSTs in the receiving system).

Expected behavior

We expect the target URL to only be called with a POST.

Version

{
    "Version": "v2.8.4+c279299",
    "BuildDate": "2023-09-13T19:12:09Z",
    "GitCommit": "c27929928104dc37b937764baf65f38b78930e59",
    "GitTreeState": "clean",
    "GoVersion": "go1.20.6",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.1.0 2023-06-19T16:58:18Z",
    "HelmVersion": "v3.12.1+gf32a527",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0"
}

Logs

Failed to notify recipient {datadog-events } defined in resource argocd/ingress-preprod-ap-aws-1: request to { GET https://api.datadoghq.com/api/v1/events datadog-events} has failed with error code 403 : {"errors":["Forbidden"]}

[rafalotufo]

I am facing the same issue. Have only configured the webhook call with POST, but being sent as a GET.

[jharris-tc]

facing the same issue

[matias-gon]

Same here. Configuring Discord notifications with webhook. Every request is made with GET which is not supported by Discord.

[Kamran-khosravi]

I am facing the same issue.

[wiktorkisielewski]

Same here!

[toini]

We ran into this issue as well.

[toini]

I noticed the new template i had created for sending webhooks was not being used at all (thus notification was being sent as GET by default).

Moving the webhook template in to a previously existing template we had for slack fixed the situation. Webhooks are now sent as POST.

  notifiers:
    service.slack: |
      token: $slack-token
      signingSecret: $slack-signing-secret

    service.webhook.dispatcher: |
      url: http://snip
      headers:
      - name: x-caller
        value: prod-argocd

  templates:
   template.app-deployed: |
      slack:
        attachments: |
          [
            {
              "title": "Application is synced and healthy {{ .app.metadata.name}}",
              "title_link":"{{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
              "color": "#18be52",
              "fields": [
                {
                  "title": "Sync Status",
                  "value": "{{.app.status.sync.status}}",
                  "short": true
                },
                {
                  "title": "Revision",
                  "value": "{{.app.status.sync.revision}}",
                  "short": true
                }
                {{range $index, $c := .app.status.conditions}}
                {{if not $index}},{{end}}
                {{if $index}},{{end}}
                {
                  "title": "{{$c.type}}",
                  "value": "{{$c.message}}",
                  "short": true
                }
                {{end}}
              ]
            }
          ]
      webhook:
        dispatcher:
          method: POST
          path: /webhook
          body: |
            {
              "app": {{toJson .app}},
              "commit": {{toJson (call .repo.GetCommitMetadata .app.status.sync.revision)}}
            }

  triggers:
    trigger.on-deployed: |
      - description: Application is synced and healthy.
        oncePer: app.status.sync.revision
        send:
          - app-deployed
        when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'

    defaultTriggers: |
      - on-deployed

    subscriptions: |
      - recipients:
          - dispatcher
        triggers:
          - on-deployed