Victorops message_type not being honoured

[JackTHG]

What did you do?

I created a firing alert and configured an alert manager config to route alerts to Victorops with the Victorops config field message_type = warning. I am using the Victorops REST API integration.

What did you expect to see?

I expected to see the message come through to VictorOps with message_type = warning.

What did you see instead? Under which circumstances?

Instead, the message came through to VictorOps with message_type = CRITICAL.

Environment

• System information:

  insert output of uname -srm here

• Alertmanager version:

  0.25.0

• Prometheus version:

  0.42.0

• Alertmanager configuration file:

  
  global:
  resolve_timeout: 5m
  http_config:
  follow_redirects: true
  enable_http2: true
  smtp_hello: localhost
  smtp_require_tls: true
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
  telegram_api_url: https://api.telegram.org
  webex_api_url: https://webexapis.com/v1/messages
  route:
  receiver: "null"
  group_by:
  - namespace
  continue: false
  routes:
  - receiver: observability/alertmanager-procurement-config/procure-dev-receiver
  group_by:
  - alertname
  - namespace
  continue: true
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  - receiver: "null"
  matchers:
  - alertname=~"InfoInhibitor|Watchdog"
  continue: false
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  inhibit_rules:

• source_matchers:

  • severity="critical" target_matchers:
  • severity=~"warning|info" equal:
  • namespace
  • alertname

• source_matchers:

  • severity="warning" target_matchers:
  • severity="info" equal:
  • namespace
  • alertname

• source_matchers:

  • alertname="InfoInhibitor" target_matchers:
  • severity="info" equal:
  • namespace receivers:

• name: "null"

• name: observability/alertmanager-procurement-config/procure-dev-receiver victorops_configs:

  • send_resolved: true http_config: follow_redirects: true enable_http2: true api_key: api_url: https://alert.victorops.com/integrations/generic/20131114/alert/ routing_key: procure-dev message_type: info state_message: | {{ range .Alerts }} {{ .Annotations.severity }} {{ .Annotations.description }} {{ end }} entity_display_name: '{{ .CommonAnnotations.summary }}' monitoring_tool: '{{ template "victorops.default.monitoring_tool" . }}' custom_fields: annotations: '{{ range $key, $value := .CommonAnnotations }}{{ $key }}={{ $value }}{{ end }}' cluster_name: '{{ .CommonLabels.cluster_name }}' labels: '{{ range $key, $value := .CommonLabels }}{{ $key }}={{ $value }}{{ end }}' namespace: '{{ .CommonLabels.namespace }}' organisation: '{{ .CommonLabels.organization }}' pod: '{{ .CommonLabels.pod }}' service: '{{ .CommonLabels.service }}' templates:

• /etc/alertmanager/config/*.tmpl

• Prometheus configuration file:

  
  global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
  cluster_name: eic5b
  organization: sw-procurement
  prometheus: observability/prometheus-operator-promet-prometheus
  prometheus_replica: prometheus-prometheus-operator-promet-prometheus-0
  alerting:
  alert_relabel_configs:
  - separator: ;
  regex: prometheus_replica
  replacement: $1
  action: labeldrop
  alertmanagers:
  - follow_redirects: true
  enable_http2: true
  scheme: http
  path_prefix: /
  timeout: 10s
  api_version: v2
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: prometheus-operator-promet-alertmanager
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http-web
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - observability
  rule_files:

• /etc/prometheus/rules/prometheus-prometheus-operator-promet-prometheus-rulefiles-0/*.yaml scrape_configs:

• job_name: serviceMonitor/observability/prometheus-operator-promet-kube-etcd/0 honor_timestamps: true scrape_interval: 30s scrape_timeout: 10s metrics_path: /metrics scheme: http authorization: type: Bearer credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token follow_redirects: true enable_http2: true relabel_configs:

  • source_labels: [job] separator: ; regex: (.*) target_label: __tmp_prometheus_job_name replacement: $1 action: replace
  • source_labels: [meta_kubernetes_service_label_app, meta_kubernetes_service_labelpresent_app] separator: ; regex: (prometheus-operator-app-kube-etcd);true replacement: $1 action: keep
  • source_labels: [meta_kubernetes_service_label_release, meta_kubernetes_service_labelpresent_release] separator: ; regex: (prometheus-operator);true replacement: $1 action: keep
  • source_labels: [__meta_kubernetes_endpoint_port_name] separator: ; regex: http-metrics replacement: $1 action: keep
  • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name] separator: ; regex: Node;(.*) target_label: node replacement: ${1} action: replace
  • source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name] separator: ; regex: Pod;(.*) target_label: pod replacement: ${1} action: replace
  • source_labels: [__meta_kubernetes_namespace] separator: ; regex: (.*) target_label: namespace replacement: $1 action: replace
  • source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: service replacement: $1 action: replace
  • source_labels: [__meta_kubernetes_pod_name] separator: ; regex: (.*) target_label: pod replacement: $1 action: replace
  • source_labels: [__meta_kubernetes_pod_container_name] separator: ; regex: (.*) target_label: container replacement: $1 action: replace
  • source_labels: [__meta_kubernetes_pod_phase] separator: ; regex: (Failed|Succeeded) replacement: $1 action: drop
  • source_labels: [__meta_kubernetes_service_name] separator: ; regex: (.*) target_label: job replacement: ${1} action: replace
  • source_labels: [__meta_kubernetes_service_label_jobLabel] separator: ; regex: (.+) target_label: job replacement: ${1} action: replace
  • separator: ; regex: (.*) target_label: endpoint replacement: http-metrics action: replace
  • separator: ; regex: (.*) target_label: app replacement: etcd action: replace
  • source_labels: [address] separator: ; regex: (.*) modulus: 1 target_label: __tmp_hash replacement: $1 action: hashmod
  • source_labels: [__tmp_hash] separator: ; regex: "0" replacement: $1 action: keep kubernetes_sd_configs:
  • role: endpoints kubeconfig_file: "" follow_redirects: true enable_http2: true namespaces: own_namespace: false names:
    • kube-system

• Logs: n/a

[JackTHG]

This was an issue with the kube prometheus stack's default rules using lowercase severity values instead of uppercase, AM appears to ignore the value 'warning', instead 'WARNING' must be used.