multiple resolved alerts

[lcasi]

What did you do? Exec flow script to send alerts to alertmanager every 1m

alerts='[
  {
    "labels": {
        "instance": "10.70.58.29:9100",
    "alertname": "DiskRunningFull",
        "instance_name": "host1"
    },
    "annotations": {
        "trigger_value": "0"
    },
    "startsAt": "2021-10-08T07:13:40.342408949Z",
    "endsAt": "2021-10-09T07:14:40.342408949Z"
},
{
    "labels": {
        "alertname": "DiskRunningFull",
    "instance": "10.70.58.30:9100",
        "instance_name": "host2"
    },
    "annotations": {
        "trigger_value": "0"
    },
    "startsAt": "2021-10-08T07:13:40.342408949Z",
    "endsAt": "2021-10-08T07:14:40.342408949Z"
}]'
curl -XPOST -d"$alerts" http://localhost:9093/api/v1/alerts

What did you expect to see? Only one resolved alert for host2.

What did you see instead? Under which circumstances? Received multiple resolved alerts for host2 every 1m

Environment

• System information:

Linux 4.15.0-142-generic x86_64

• Alertmanager version:

0.19.0 & 0.23.0

• Alertmanager configuration file:

  
  global:
  resolve_timeout: 2m

route: group_by: ['alertname'] group_wait: 10s group_interval: 10s repeat_interval: 1m receiver: 'web.hook' receivers:

• name: 'web.hook' webhook_configs:

  • url: 'http://127.0.0.1:5001/' inhibit_rules:
  • source_match: severity: 'critical' target_match: severity: 'warning' equal: ['alertname', 'dev', 'instance']```

• Prometheus configuration file:

  insert configuration here (if relevant to the issue)

• Logs: no logs output

[erachenko23]

We have this problem too on 0.23.

After restarting alertmanager problem is solved, but after sometime back. Usualy problem started after mass alarms generation.