SNS Topic Arn not parsing correctly

[rrraikman]

What did you do? Configured an SNS receiver

What did you expect to see? An alert be sent

What did you see instead? Under which circumstances? Seeing the following logs in all alert circumstances:

ts=2024-11-05T07:35:10.350Z caller=dispatch.go:353 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="sns/sns[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: Invalid parameter: TopicArn Reason: An ARN must have at least 6 elements, not 1"

Environment Kubernetes/EKS

• System information:

Linux 6.1.112-122.189.amzn2023.aarch64 aarch64

• Alertmanager version:

Seeing this on 0.27.0, 0.26.0, 0.25.0, and 0.24.0 as well.

alertmanager, version 0.27.0 (branch: HEAD, revision: 0aa3c2aad14cff039931923ab16b26b7481783b5)
  build user:       root@2024b1e0f6e3
  build date:       20240228-11:47:50
  go version:       go1.21.7
  platform:         linux/arm64
  tags:             netgo

• Prometheus version:

  prometheus, version 2.54.1 (branch: HEAD, revision: e6cfa720fbe6280153fab13090a483dbd40bece3)
  build user:       root@812ffd741951
  build date:       20240827-10:59:03
  go version:       go1.22.6
  platform:         linux/arm64
  tags:             netgo,builtinassets,stringlabels

• Alertmanager configuration file:

    route:
      group_by: ["alertname"]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: sns
      routes:
        - match:
            alertname: Watchdog
          group_wait: 30s
          group_interval: 1m
          repeat_interval: 1m
          receiver: sns
    receivers:
      - name: sns
        sns_configs:
          - topic_arn: arn:aws:sns:us-east-2:<account-id-redacted>:<topic-name-redacted>
            attributes:
              Key: severity
              Value: "{{ .Labels.severity}}"
            message: |
              Alert: {{ .Annotations.summary }}
              Severity: {{ .Labels.severity }}
              Description: {{ .Labels.description }}
            subject: "[Prometheus Alert] {{ .Labels.alertname }}"
            send_resolved: true

I have also tried with the following configuration file:

    route:
      group_by: ["alertname"]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: sns
      routes:
        - match:
            alertname: Watchdog
          group_wait: 30s
          group_interval: 1m
          repeat_interval: 1m
          receiver: sns
    receivers:
      - name: sns
        sns_configs:
          - api_url: https://sns.us-east-2.amazonaws.com
            topic_arn: arn:aws:sns:us-east-2:<account-id-redacted>:<topic-name-redacted>
            sigv4:
              role_arn: arn:aws:iam::<account-id-redacted>:role/<role-name-redacted>
              region: us-east-2
            attributes:
              Key: severity
              Value: "{{ .Labels.severity}}"
            message: |
              Alert: {{ .Annotations.summary }}
              Severity: {{ .Labels.severity }}
              Description: {{ .Labels.description }}
            subject: "[Prometheus Alert] {{ .Labels.alertname }}"
            send_resolved: true

• Prometheus configuration file:

N/A

• Logs:

  ts=2024-11-05T07:35:10.350Z caller=dispatch.go:353 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="sns/sns[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: Invalid parameter: TopicArn Reason: An ARN must have at least 6 elements, not 1"

[TheSwarnim]

We're also facing the same issue above

[Alok650]

Following. Facing the same issue

[rrraikman]

I think the error message is just misleading. My issue ended up being related to the use of .Labels and .Annotations. It should have been CommonAnnotations and .CommonLabels, respectively.

My new (working) configuration is:

    route:
      group_by: ["alertname"]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: sns
      routes:
        - match:
            alertname: Watchdog
          group_wait: 30s
          group_interval: 1m
          repeat_interval: 1m
          receiver: sns
    receivers:
      - name: sns
        sns_configs:
          - topic_arn: arn:aws:sns:us-east-2:<account-id-redacted>:<topic-name-redacted>
            attributes:
              Key: severity
              Value: "{{ .CommonLabels.severity }}"
            message: |
              Severity: {{ .CommonLabels.severity }}
              Summary: {{ .CommonAnnotations.summary }}
              Description: {{ .CommonAnnotations.description }}
            subject: "[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}"
            send_resolved: true