hsggj002
commented
2 years ago 这个错误是因为prometheus里面没有配置region这个label, 如果告警不是多机房或多地区可以将代码中region字段屏蔽掉。 解决方法（两种方式） 1、prometheus添加labels: region: xxx 2、修改代码进行屏蔽
Open tonglv opened 2 years ago
hsggj002
commented
2 years ago 这个错误是因为prometheus里面没有配置region这个label, 如果告警不是多机房或多地区可以将代码中region字段屏蔽掉。 解决方法（两种方式） 1、prometheus添加labels: region: xxx 2、修改代码进行屏蔽
Loaded: loaded (/usr/lib/systemd/system/prometheus-flask.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2022-10-13 14:04:19 CST; 3min 4s ago Main PID: 10321 (python3.6) Tasks: 2 Memory: 27.1M CGroup: /system.slice/prometheus-flask.service └─10321 /bin/python3.6 /apps/mon/prometheus-flask-main/app/main.py -p 5000 -k https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=d57298da-3cb1-4232-8a3c-6df9af5a72b4
Oct 13 14:07:24 bszx_zabbix python3.6[10321]: rv = self.handle_user_exception(e) Oct 13 14:07:24 bszx_zabbix python3.6[10321]: File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1513, in full_dispatch_request Oct 13 14:07:24 bszx_zabbix python3.6[10321]: rv = self.dispatch_request() Oct 13 14:07:24 bszx_zabbix python3.6[10321]: File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1499, in dispatch_request Oct 13 14:07:24 bszx_zabbix python3.6[10321]: return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Oct 13 14:07:24 bszx_zabbix python3.6[10321]: File "/apps/mon/prometheus-flask-main/app/main.py", line 18, in alert_data Oct 13 14:07:24 bszx_zabbix python3.6[10321]: Alert.send_alert(json_re, args.key) Oct 13 14:07:24 bszx_zabbix python3.6[10321]: File "/apps/mon/prometheus-flask-main/app/Alert.py", line 52, in send_alert Oct 13 14:07:24 bszx_zabbix python3.6[10321]: webhook_url(alert(i['status'],i['labels']['region'],i['labels']['alertname'],i['labels']['severity'],parse_time(i['startsAt']),'故障实例',i['labels']['instance'],i['annotations']['description']),url_key) Oct 13 14:07:24 bszx_zabbix python3.6[10321]: KeyError: 'region'
alertmanager.yml配置
Oct 13 14:09:12 bszx_zabbix alertmanager[4662]: level=error ts=2022-10-13T06:09:12.636Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="webhook/webhook[0]: notify retry canceled after 7 attempts: unexpected status code 500: http://localhost:5000/alertinfo" Oct 13 14:09:12 bszx_zabbix alertmanager[4662]: level=warn ts=2022-10-13T06:09:12.638Z caller=notify.go:724 component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: http://localhost:5000/alertinfo" Oct 13 14:09:15 bszx_zabbix alertmanager[4662]: level=error ts=2022-10-13T06:09:15.505Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="webhook/webhook[0]: notify retry canceled after 8 attempts: unexpected status code 500: http://localhost:5000/alertinfo" Oct 13 14:09:15 bszx_zabbix alertmanager[4662]: level=warn ts=2022-10-13T06:09:15.507Z caller=notify.go:724 component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: http://localhost:5000/alertinfo" Oct 13 14:09:18 bszx_zabbix alertmanager[4662]: level=error ts=2022-10-13T06:09:18.446Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="webhook/webhook[0]: notify retry canceled after 7 attempts: unexpected status code 500: http://localhost:5000/alertinfo" Oct 13 14:09:18 bszx_zabbix alertmanager[4662]: level=warn ts=2022-10-13T06:09:18.448Z caller=notify.go:724 component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: http://localhost:5000/alertinfo" Oct 13 14:09:19 bszx_zabbix alertmanager[4662]: level=error ts=2022-10-13T06:09:19.554Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="webhook/webhook[0]: notify retry canceled after 7 attempts: unexpected status code 500: http://localhost:5000/alertinfo" Oct 13 14:09:19 bszx_zabbix alertmanager[4662]: level=warn ts=2022-10-13T06:09:19.556Z caller=notify.go:724 component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: http://localhost:5000/alertinfo" Oct 13 14:09:22 bszx_zabbix alertmanager[4662]: level=error ts=2022-10-13T06:09:22.637Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="webhook/webhook[0]: notify retry canceled after 8 attempts: unexpected status code 500: http://localhost:5000/alertinfo" Oct 13 14:09:22 bszx_zabbix alertmanager[4662]: level=warn ts=2022-10-13T06:09:22.639Z caller=notify.go:724 component=dispatcher receiver=webhook integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="unexpected status code 500: http://localhost:5000/alertinfo"