canonical / alertmanager-k8s-operator

This charmed operator automates operation procedures of Alertmanager, the alerting component of Prometheus and Loki, among others.
https://charmhub.io/alertmanager-k8s
Apache License 2.0
4 stars 16 forks source link

Don't use non-existent webhook as a placeholder #238

Closed nobuto-m closed 3 months ago

nobuto-m commented 3 months ago

Issue

Fixes: #237

Solution

The fake webhook fires the AlertmanagerNotificationsFailed alert continuously out of the box. Instead, use a blackhole receiver without any backend as a placeholder when no receiver is provided by an operator.

Context

[before patching]

global:
  http_config:
    tls_config:
      insecure_skip_verify: false
receivers:
- name: placeholder
  webhook_configs:
  - url: http://127.0.0.1:5001/
route:
  group_by:
  - juju_model
  - juju_application
  - juju_model_uuid
  group_interval: 5m
  group_wait: 30s
  receiver: placeholder
  repeat_interval: 1h

[after patching]

global:
  http_config:
    tls_config:
      insecure_skip_verify: false
receivers:
- name: placeholder
route:
  group_by:
  - juju_application
  - juju_model
  - juju_model_uuid
  group_interval: 5m
  group_wait: 30s
  receiver: placeholder
  repeat_interval: 1h

[After patching] Screenshot from 2024-03-19 12-00-18

[The fix was applied in the middle of this timeline] Screenshot from 2024-03-19 11-56-56

Testing Instructions

The steps are in #237.

  1. deploy COS and workload
  2. related COS with workload
  3. check alerts

Upgrade Notes

N/A

nobuto-m commented 3 months ago

More context: