prometheus / alertmanager

Prometheus Alertmanager
https://prometheus.io
Apache License 2.0
6.64k stars 2.15k forks source link

Silence regex match behaving unintuitively #3279

Open sinkingpoint opened 1 year ago

sinkingpoint commented 1 year ago

What did you do?

Created a silence with a regex matcher with a double quote in it

What did you expect to see?

The silence shouldn't match anything

What did you see instead? Under which circumstances?

The silence matches everything

Environment

/alertmanager $ alertmanager --version
alertmanager, version 0.25.0 (branch: HEAD, revision: 258fab7cdd551f2cf251ed0348f0ad7289aee789)
build user:       root@abe866dd5717
build date:       20221222-14:51:36
go version:       go1.19.4
platform:         linux/amd64

The default from prom/alertmanager:v0.25.0

global:
  resolve_timeout: 5m
  http_config:
    follow_redirects: true
    enable_http2: true
  smtp_hello: localhost
  smtp_require_tls: true
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
  telegram_api_url: https://api.telegram.org
  webex_api_url: https://webexapis.com/v1/messages
route:
  receiver: web.hook
  group_by:
  - alertname
  continue: false
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
inhibit_rules:
- source_match:
    severity: critical
  target_match:
    severity: warning
  equal:
  - alertname
  - dev
  - instance
receivers:
- name: web.hook
  webhook_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    url: http://127.0.0.1:5001/
    max_alerts: 0
templates: []

Bit of a long form description for replication:

  1. Start an alertmanager: docker run -d -p 9093:9093 prom/alertmanager:v0.25.0
  2. Send an alert with a single label, instance=foo: curl -XPOST localhost:9093/api/v2/alerts -H 'Content-Type: application/json' -d '[{"labels":{"instance":"foo"}}]'
  3. Create a silence with a single matcher, instance=~"bar": curl -XPOST localhost:9093/api/v2/silences -H 'Content-Type: application/json' -d '{"comment":"foo", "createdBy": "cd@cloudflare", "startsAt":"2023-03-06T00:22:15Z", "endsAt":"2024-03-06T00:22:15Z", "matchers":[{"isRegex":true, "name":"instance", "value":"\"bar\""}]}'

Observe that the created silence matches the alert: image

Note that the double quotes are included in the regex erroneously - I got confused because the Web UI enforces the use of quotes around matchers and strips them, but the API does not do this. But either way, including quotes in the regex should not match anything that doesn't also have quotes in it.

sinkingpoint commented 1 year ago

Looking at this a bit deeper, it might be a UI bug. Opening the silence page for a silence without the quotes sends a request to: http://localhost:9093/api/v2/alerts?filter=instance%3D~%22bar%22&silenced=true&inhibited=true&active=true, whereas opening the silence page for the silence with quotes sends a request to http://localhost:9093/api/v2/alerts?silenced=true&inhibited=true&active=true (without a filter), but I don't know enough elm to confirm this