nabilrad
commented
3 years ago We are seeing the same scenario as we are getting false alarms every time based on repeat_interval value and no events are getting record on both alertamanger and prometheus ui.
$./alertmanager --version alertmanager, version 0.23.0 (branch: HEAD, revision: 61046b17771a57cfd4c4a51be370ab930a4d7d54) build user: root@e21a959be8d2 build date: 20210825-10:48:55 go version: go1.16.7
$ ./prometheus --version prometheus, version 2.26.0 (branch: HEAD, revision: 3cafc58827d1ebd1a67749f88be4218f0bab3d8d) build user: root@a67cafebe6d0 build date: 20210331-11:56:23 go version: go1.16.2 platform: linux/amd64
$ uname -a Linux seigwpdevmon01 5.4.17-2102.205.7.3.el7uek.x86_64 #2 SMP Fri Sep 17 16:52:13 PDT 2021 x86_64 x86_64 x86_64 GNU/Linux
roidelapluie
23ewrdtf
xenofree
mutagenspree
simonpasquier
MMGeri
What did you do? I installed alert manager with prometheus using this chart: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
prometheus-community:Chart:14.10.0:App:2.26.0What did you expect to see? I have an alert in Prometheus:
When I go to https://prometheus/alerts this alert is not there.
When I go to https://alertmanager/#/alerts this alert is not there.
When I go to
https://prometheus/graph?g0.expr=up%20%3D%3D%200&g0.tab=1&g0.stacked=0&g0.range_input=1hI don't get any resultsI get below alert to my slack every 5 minutes (together with other current alerts):
AlertManagerAPP 12:41 [FIRING:21] Monitoring Event Notification Alert: Prometheus target missing xxx - critical Description: A Prometheus target has disappeared. An exporter might be crashed. `` Details: • alertname: PrometheusTargetMissing • endpoint: metrics Show more
[FIRING:25] Monitoring Event Notification 12:46 Alert: Prometheus target missing xxx - critical Description: A Prometheus target has disappeared. An exporter might be crashed. `` Details: • alertname: PrometheusTargetMissing • endpoint: metrics Show more
/alertmanager $ uname -srm Linux 4.14.243-185.433.amzn2.x86_64 x86_64
/alertmanager $ alertmanager --version alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d) build user: root@dee35927357f build date: 20200617-08:54:02 go version: go1.14.4
/prometheus $ prometheus --version prometheus, version 2.26.0 (branch: HEAD, revision: 3cafc58827d1ebd1a67749f88be4218f0bab3d8d) build user: root@a67cafebe6d0 build date: 20210331-11:56:23 go version: go1.16.2 platform: linux/amd64
global: resolve_timeout: 5m http_config: {} smtp_hello: localhost smtp_require_tls: true slack_api_url:
pagerduty_url: xxx
opsgenie_api_url: xxx
wechat_api_url: xxx
victorops_api_url: xxx
route:
receiver: slack
group_wait: 30s
group_interval: 5m
repeat_interval: 3h
receivers:
name: slack slack_configs:
{{ .Labels.severity }}Description: {{ .Annotations.description }} Details: {{ range .Labels.SortedPairs }} • {{ .Name }}:{{ .Value }}{{ end }} {{ end }} footer: '{{ template "slack.default.footer" . }}' fallback: '{{ template "slack.default.fallback" . }}' callback_id: '{{ template "slack.default.callbackid" . }}' icon_emoji: '{{ template "slack.default.iconemoji" . }}' icon_url: https://avatars3.githubusercontent.com/u/3380462 templates: []Prometheus configuration file:
Logs: