time_interval definition is not taken to be in UTC

[zakaf]

What did you do?

• according to the official documentation, all time_interval definitions are taken to be in UTC
• set mute_time_interval in UTC and added the interval to certain routing rule

  route:
  group_by: [ 'severity', 'alertname', 'job' ]
  group_wait: 30s
  receiver: 'slack.alert'
  routes:
  - matchers: ["severity=warning"]
    group_interval: 5m
    group_wait: 30s
    repeat_interval: 10m
    mute_time_intervals:
      - off_hours
  ...
  mute_time_intervals:
  - name: off_hours
  time_intervals:
    - times:
      - start_time: 13:00 # KST 22:00
        end_time: 24:00 # KST 09:00

What did you expect to see?

• mute alerts within the specified time range (13:00 ~ 24:00 UTC)

What did you see instead? Under which circumstances?

• received alerts within the specified time range
• if i put the time interval in my system's timezone (KST), it mutes as expected

Environment

• System information: Linux 3.10.0-1062.12.1.el7.x86_64 x86_64 $ date +"%Z %z -> KST +0900

• Alertmanager version:

  alertmanager, version 0.22.2 (branch: HEAD, revision: 44f8adc06af5101ad64bd8b9c8b18273f2922051)
  build user:       root@b595c7f32520
  build date:       20210602-07:50:37
  go version:       go1.16.4
  platform:         linux/amd64

• Prometheus version:

  prometheus, version 2.27.1 (branch: HEAD, revision: db7f0bcec27bd8aeebad6b08ac849516efa9ae02)
  build user:       root@fd804fbd4f25
  build date:       20210518-14:17:54
  go version:       go1.16.4
  platform:         linux/amd64

• Alertmanager configuration file:

  
  global:
  http_config:
  proxy_url: '*****'

templates:

• 'monzo-slack-templates.tmpl'

The root node of the routing tree.

route: group_by: [ 'severity', 'alertname', 'job' ] group_wait: 30s receiver: 'slack.alert' routes:

• matchers: ["severity=critical"] repeat_interval: 1m group_wait: 5s group_interval: 30s
• matchers: ["severity=error"] repeat_interval: 5m group_wait: 30s group_interval: 1m
• matchers: ["severity=warning"] group_interval: 5m group_wait: 30s repeat_interval: 10m mute_time_intervals:
  • off_hours

A list of notification receivers.

receivers:

• name: 'slack.alert' slack_configs:
  • api_url: '****' send_resolved: true title: '{{ template "slack.monzo.title" . }}' color: '{{ template "slack.monzo.color" . }}' text: '{{ template "slack.monzo.text" . }}' actions:
    • type: button text: 'Runbook :green_book:' url: '{{ (index .Alerts 0).Annotations.runbook }}'
    • type: button text: 'Query :mag:' url: '{{ (index .Alerts 0).GeneratorURL }}'
    • type: button text: 'Silence :no_bell:' url: '{{ template "__alert_silence_link" . }}'

A list of mute time intervals for muting routes.

mute_time_intervals:

• name: off_hours time_intervals:
  • times:
    • start_time: 13:00 # KST 22:00 end_time: 24:00 # KST 09:00

• Prometheus configuration file:

  insert configuration here (if relevant to the issue)

• Logs: i've set the log level to debug for test purposes and the below logs showed up with different time interval test

  
  // set mute_time_interval as 12:00~24:00
  7월 12 12:00:30 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:00:30.168Z caller=dispatch.go:475 component=dispatcher aggrGroup="{}/{severity=\"warning\"}:{alertname=\"AboveAverageConnectionCnt\", job=\"k8s\", severity=\"warning\"}" msg=flushing alerts="[AboveAverageConnectionCnt[ff9598c][active] AboveAverageConnectionCnt[ed30b21][active] AboveAverageConnectionCnt[4639a49][active]]"
  7월 12 12:00:30 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:00:30.597Z caller=cluster.go:693 component=cluster msg="gossip looks settled" elapsed=6.00270398s
  7월 12 12:00:32 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:00:32.598Z caller=cluster.go:693 component=cluster msg="gossip looks settled" elapsed=8.003836146s
  7월 12 12:00:34 host.address.com alertmanager[3251]: level=info ts=2021-07-12T03:00:34.599Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.004016694s
  7월 12 12:00:34 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:00:34.600Z caller=notify.go:818 component=dispatcher msg="Notifications not sent, route is within mute time"
  7월 12 12:00:34 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:00:34.600Z caller=dispatch.go:475 component=dispatcher aggrGroup="{}/{severity=\"warning\"}:{alertname=\"AboveAverageConnectionCnt\", job=\"k8s\", severity=\"warning\"}" msg=flushing alerts="[AboveAverageConnectionCnt[ff9598c][active] AboveAverageConnectionCnt[ed30b21][active] AboveAverageConnectionCnt[4639a49][active]]"
  7월 12 12:00:34 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:00:34.600Z caller=notify.go:818 component=dispatcher msg="Notifications not sent, route is within mute time"
  7월 12 12:01:45 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:01:45.151Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[ff9598c][active]
  7월 12 12:01:45 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:01:45.151Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[ed30b21][active]
  7월 12 12:01:45 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:01:45.151Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[4639a49][active]
  7월 12 12:03:00 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:03:00.151Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[ff9598c][active]
  7월 12 12:03:00 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:03:00.151Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[ed30b21][active]
  7월 12 12:03:00 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:03:00.151Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[4639a49][active]
  7월 12 12:04:15 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:04:15.152Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[ff9598c][active]
  7월 12 12:04:15 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:04:15.153Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[ed30b21][active]
  7월 12 12:04:15 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:04:15.153Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[4639a49][active]
  7월 12 12:05:30 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:05:30.148Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[ff9598c][active]
  7월 12 12:05:30 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:05:30.148Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[ed30b21][active]
  7월 12 12:05:30 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:05:30.148Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[4639a49][active]
  7월 12 12:05:34 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:05:34.601Z caller=dispatch.go:475 component=dispatcher aggrGroup="{}/{severity=\"warning\"}:{alertname=\"AboveAverageConnectionCnt\", job=\"k8s\", severity=\"warning\"}" msg=flushing alerts="[AboveAverageConnectionCnt[ff9598c][active] AboveAverageConnectionCnt[ed30b21][active] AboveAverageConnectionCnt[4639a49][active]]"
  7월 12 12:05:34 host.address.com alertmanager[3251]: level=debug ts=2021-07-12T03:05:34.602Z caller=notify.go:818 component=dispatcher msg="Notifications not sent, route is within mute time"

// set mute_time_interval as 13:00~24:00 7월 12 12:08:00 host.address.com alertmanager[6123]: level=debug ts=2021-07-12T03:08:00.149Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[ff9598c][active] 7월 12 12:08:00 host.address.com alertmanager[6123]: level=debug ts=2021-07-12T03:08:00.149Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[ed30b21][active] 7월 12 12:08:00 host.address.com alertmanager[6123]: level=debug ts=2021-07-12T03:08:00.149Z caller=dispatch.go:138 component=dispatcher msg="Received alert" alert=AboveAverageConnectionCnt[4639a49][active] 7월 12 12:08:00 host.address.com alertmanager[6123]: level=debug ts=2021-07-12T03:08:00.150Z caller=dispatch.go:475 component=dispatcher aggrGroup="{}/{severity=\"warning\"}:{alertname=\"AboveAverageConnectionCnt\", job=\"k8s\", severity=\"warning\"}" msg=flushing alerts="[AboveAverageConnectionCnt[ff9598c][active] AboveAverageConnectionCnt[ed30b21][active] AboveAverageConnectionCnt[4639a49][active]]" 7월 12 12:08:00 host.address.com alertmanager[6123]: level=debug ts=2021-07-12T03:08:00.915Z caller=notify.go:734 component=dispatcher receiver=slack.alert integration=slack[0] msg="Notify success" attempts=1

[roidelapluie]

cc @benridley can you please have a look at this report? Thanks!

[benridley]

Thanks @zakaf for raising and @roidelapluie for bringing this to my attention.

The mute time mechanism currently uses the aggregation group's internal timer to determine the current time and make comparisons with mute times. Turns out this timer fires in local time, not in UTC. I've raised #2648 to ensure comparisons are only made in UTC and added a few test cases to confirm.

[zakaf]

@roidelapluie @benridley Thanks for the bug fix :) I've tried out 0.23.0 and the #2648 has fixed the issue