GroupWait and Interval 0 causes a lot of allocs and logs

[gouthamve]

What did you do? Ran a single AM instance with GroupWait and Interval 0 and debug logging.

What did you expect to see? Chatty but useful logs.

What did you see instead? Under which circumstances? A huge flood!

• Alertmanager version: alertmanager, version 0.11.0 (branch: HEAD, revision: 30dd0426c08b6479d9a26259ea5efd63bc1ee273) build user: root@3e103e3fc918 build date: 20171116-17:43:56 go version: go1.9.2

• Alertmanager configuration file:

  
  # The root route on which each incoming alert enters.
  route:
  # The labels by which incoming alerts are grouped together. For example,
  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  # be batched into a single group.
  group_by: ['alertname', 'job']
  
  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first 
  # notification.
  group_wait: 30s
  
  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 5m
  
  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 1h 
  
  # All the above attributes are inherited by all child routes and can 
  # overwritten on each.
  receiver: default-receiver 
  
  # The child route trees.
  routes:
  - receiver: pushover
  match:
    severity: critical
  - receiver: deadmans-switch
  group_wait: 0s
  group_interval: 0s
  repeat_interval: 15s
  match:
    severity: deadman

receivers:

• name: 'pushover' pushover_configs:
  • user_key: token:
• name: deadmans-switch webhook_configs:
  • url: http://deadman:9095
• name: default-receiver

* Prometheus configuration file:

insert configuration here (if relevant to the issue)

* Logs:

level=debug ts=2017-11-20T17:12:00.225274251Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.225427596Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.225531945Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.225607822Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.225677089Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.225736667Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.225795896Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.225871146Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.225968086Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.226105244Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.22625187Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.226384052Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.226514799Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.226650737Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.226782719Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.226873043Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.226945495Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.227016107Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.22708717Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.22715533Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.22722366Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.227304341Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.227421041Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.227556633Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.227703011Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.227852496Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.227989343Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.228120466Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]] level=debug ts=2017-11-20T17:12:00.228249764Z caller=dispatch.go:430 component=dispatcher aggrGroup="{}/{severity=\"deadman\"}:{alertname=\"DeadManBoy\"}" msg=Flushing alerts=[DeadManBoy[8ba1e9d][active]]

This is a super small subset.

[gouthamve]

Sorry. Config issue on my end. I put both group_wait and group_interval as 0.

Not sure if that should cause a barrage of notifications though. We can atleast protect this case by putting a check.

Feel free to close if this doesn't make sense.

[gouthamve]

So notifications were being sent every repeat_interval only. But this loop being run so tightly is causing a lot of allocs: rate(go_memstats_alloc_bytes_total) = 60M

[fabxc]

People usually set too low intervals. But 0 is just 100% wrong semantically. High resource usage is more than expected in this case.

We should probably return a configuration error in this case.

[simonpasquier]

@fabxc IIUC this is similar to #583.

In the comments, another proposal was to have a sane minimum value (eg 1s) in case the parameter was zero. Personally I'd prefer a hard failure (explicit over implicit) but wanting to check.

[fabxc]

Yes, a hard failure is probably better so that there are no wrong expectations.

[simonpasquier]

Thanks! I'll give it a shot then

[puneets-ampere]

if i am interested only in repeat_interval , and do not need group_interval, how do i express this via configurations? maybe group_interval=0 could have helped in this scenario?