slack notify retry canceled due to unrecoverable error 403

[johirulalam]

ts=2024-02-18T18:53:00.366Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="slack-notifications/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"#monitoring\": unexpected status code 403: invalid_token"

alertmanager.yml global: slack_api_url: 'https://hooks.slack.com/services/T06L1G****'

route: receiver: 'slack-notifications'

receivers:

• name: 'slack-notifications' slack_configs:
  • channel: '#monitoring'

Rules

groups:

• name: os rules:

  • alert: NodeExporterDown expr: up{job="node_exporter"} == 0 for: 15s labels: severity: critical annotations: summary: "Node Exporter is down" description: "Node Exporter is down for more than 2 minutes"

[rapphil]

Hi, 403 error codes should not be retried. IT seems that there is some issue with your slack configuration: https://api.slack.com/messaging/webhooks#handling_errors

[ktnvaish22]

Hi @rapphil Does alertmanager retry only 5xx errors? Is there a documentation present on retry mechanism of Alertmanager?