AngleMAXIN commented 4 years ago

钉钉报警没效果，我看了打印的日志 PostToDingDing:Post : unsupported protocol scheme "" app.conf

#是否开启钉钉告警通道,可同时开始多个通道0为关闭,1为开启
    open-dingding=1
    #默认钉钉机器人地址
    ddurl=https://oapi.dingtalk.com/robot/send?access_token=xxxxxxx

leon0306 commented 4 years ago

这个问题你是怎么解决的?

python-it commented 3 years ago

我这边也报错：（只开启了飞书报警） 2021/01/12 07:25:05.105 [E] [prometheus.go:149] [1610436305105833601] [dingding] Post "": unsupported protocol scheme "" 2021/01/12 07:25:05.106 [C] [panic.go:969] the request url is /prometheus/alert 2021/01/12 07:25:05.106 [C] [panic.go:969] Handler crashed with error runtime error: invalid memory address or nil pointer dereference 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/panic.go:969 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/panic.go:212 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/signal_unix.go:717 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/dingding.go:71 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/prometheus.go:149 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/prometheus.go:67 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/reflect/value.go:460 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/reflect/value.go:321 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/vendor/github.com/astaxie/beego/router.go:853 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/net/http/server.go:2836 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/net/http/server.go:1924 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/asm_amd64.s:1373 2021/01/12 07:25:05.106 [server.go:3088] [HTTP] http: superfluous response.WriteHeader call from github.com/astaxie/beego/context.(*Response).WriteHeader (context.go:230)

feiyu563 commented 3 years ago

注意app.conf文件中的其他配置项不需要的切勿删除，保留默认即可，删除了的话就会造成你上面的报错

---原始邮件--- 发件人: "python-it"<notifications@github.com> 发送时间: 2021年1月12日(周二) 下午3:34 收件人: "feiyu563/PrometheusAlert"<PrometheusAlert@noreply.github.com>; 抄送: "Subscribed"<subscribed@noreply.github.com>; 主题: Re: [feiyu563/PrometheusAlert] 钉钉报警失效，不支持协议 (#6)

我这边也报错：（只开启了飞书报警） 2021/01/12 07:25:05.105 [E] [prometheus.go:149] [1610436305105833601] [dingding] Post "": unsupported protocol scheme "" 2021/01/12 07:25:05.106 [C] [panic.go:969] the request url is /prometheus/alert 2021/01/12 07:25:05.106 [C] [panic.go:969] Handler crashed with error runtime error: invalid memory address or nil pointer dereference 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/panic.go:969 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/panic.go:212 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/signal_unix.go:717 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/dingding.go:71 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/prometheus.go:149 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/prometheus.go:67 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/reflect/value.go:460 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/reflect/value.go:321 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/vendor/github.com/astaxie/beego/router.go:853 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/net/http/server.go:2836 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/net/http/server.go:1924 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/asm_amd64.s:1373 2021/01/12 07:25:05.106 [server.go:3088] [HTTP] http: superfluous response.WriteHeader call from github.com/astaxie/beego/context.(*Response).WriteHeader (context.go:230)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

python-it commented 3 years ago

app.conf保持其他默认配置，还是报错了，只开启了飞书报警（prometheus-operator），求大佬指教，谢谢！ 2021/01/12 09:47:15.194 [I] [prometheus.go:168] [1610444835107573678] [weixin] 企业微信接口未配置未开启状态,请先配置open-weixin为1 2021/01/12 09:47:15.194 [I] [feishu.go:32] [1610444835107573678] [feishu] {"title":"PrometheusAlert故障告警信息","text":"## PrometheusAlertPrometheus故障告警信息\n\n#### KubeDeploymentReplicasMismatch\n\n###### 告警级别：信息\n\n###### 开始时间：2021-01-12T09:47:05.095800493Z\n\n###### 结束时间：0001-01-01T00:00:00Z\n\n###### 故障主机IP：172.17.1.106:8443\n\n##### \n\n"}

2021/01/12 09:47:15.194 [E] [feishu.go:32] [1610444835107573678] [feishu] Post "": unsupported protocol scheme "" 2021/01/12 09:47:15.194 [C] [panic.go:969] the request url is /prometheus/alert 2021/01/12 09:47:15.194 [C] [panic.go:969] Handler crashed with error runtime error: invalid memory address or nil pointer dereference 2021/01/12 09:47:15.194 [C] [panic.go:969] /usr/local/go/src/runtime/panic.go:969 2021/01/12 09:47:15.194 [C] [panic.go:969] /usr/local/go/src/runtime/panic.go:212 2021/01/12 09:47:15.194 [C] [panic.go:969] /usr/local/go/src/runtime/signal_unix.go:717 2021/01/12 09:47:15.194 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/feishu.go:62 2021/01/12 09:47:15.194 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/feishu.go:32 2021/01/12 09:47:15.194 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/prometheus.go:186 2021/01/12 09:47:15.195 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/prometheus.go:67 2021/01/12 09:47:15.195 [C] [panic.go:969] /usr/local/go/src/reflect/value.go:460 2021/01/12 09:47:15.195 [C] [panic.go:969] /usr/local/go/src/reflect/value.go:321 2021/01/12 09:47:15.195 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/vendor/github.com/astaxie/beego/router.go:853 2021/01/12 09:47:15.195 [C] [panic.go:969] /usr/local/go/src/net/http/server.go:2836 2021/01/12 09:47:15.195 [C] [panic.go:969] /usr/local/go/src/net/http/server.go:1924 2021/01/12 09:47:15.195 [C] [panic.go:969] /usr/local/go/src/runtime/asm_amd64.s:1373 2021/01/12 09:47:15.195 [server.go:3088] [HTTP] http: superfluous response.WriteHeader call from github.com/astaxie/beego/context.(*Response).WriteHeader (context.go:230)

------------------ 原始邮件 ------------------ 发件人: "feiyu563/PrometheusAlert" <notifications@github.com>; 发送时间: 2021年1月12日(星期二) 下午4:55 收件人: "feiyu563/PrometheusAlert"<PrometheusAlert@noreply.github.com>; 抄送: "暴戾ing"<164859527@qq.com>;"Comment"<comment@noreply.github.com>; 主题: Re: [feiyu563/PrometheusAlert] 钉钉报警失效，不支持协议 (#6)

注意app.conf文件中的其他配置项不需要的切勿删除，保留默认即可，删除了的话就会造成你上面的报错

---原始邮件--- 发件人: "python-it"<notifications@github.com> 发送时间: 2021年1月12日(周二) 下午3:34 收件人: "feiyu563/PrometheusAlert"<PrometheusAlert@noreply.github.com>; 抄送: "Subscribed"<subscribed@noreply.github.com>; 主题: Re: [feiyu563/PrometheusAlert] 钉钉报警失效，不支持协议 (#6)

我这边也报错：（只开启了飞书报警） 2021/01/12 07:25:05.105 [E] [prometheus.go:149] [1610436305105833601] [dingding] Post "": unsupported protocol scheme "" 2021/01/12 07:25:05.106 [C] [panic.go:969] the request url is /prometheus/alert 2021/01/12 07:25:05.106 [C] [panic.go:969] Handler crashed with error runtime error: invalid memory address or nil pointer dereference 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/panic.go:969 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/panic.go:212 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/signal_unix.go:717 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/dingding.go:71 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/prometheus.go:149 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/controllers/prometheus.go:67 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/reflect/value.go:460 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/reflect/value.go:321 2021/01/12 07:25:05.106 [C] [panic.go:969] /go/src/github.com/feiyu563/PrometheusAlert/vendor/github.com/astaxie/beego/router.go:853 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/net/http/server.go:2836 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/net/http/server.go:1924 2021/01/12 07:25:05.106 [C] [panic.go:969] /usr/local/go/src/runtime/asm_amd64.s:1373 2021/01/12 07:25:05.106 [server.go:3088] [HTTP] http: superfluous response.WriteHeader call from github.com/astaxie/beego/context.(*Response).WriteHeader (context.go:230)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

feiyu563 commented 3 years ago

你的配置文件贴下我看看

python-it commented 3 years ago

apiVersion: v1 data: app.conf: |

---------------------↓全局配置-----------------------

appname = PrometheusAlert
#监听端口
httpport = 8080
runmode = dev
#设置代理 proxy = http://123.123.123.123:8080
proxy =
#开启JSON请求
copyrequestbody = true
#告警消息标题
title=PrometheusAlert
#链接到告警平台地址
GraylogAlerturl=http://graylog.org
#钉钉告警 告警logo图标地址
logourl=https://raw.githubusercontent.com/feiyu563/PrometheusAlert/master/doc/alert-center.png
#钉钉告警 恢复logo图标地址
rlogourl=https://raw.githubusercontent.com/feiyu563/PrometheusAlert/master/doc/alert-center.png
#短信告警级别(等于3就进行短信告警) 告警级别定义 0 信息,1 警告,2 一般严重,3 严重,4 灾难
messagelevel=3
#电话告警级别(等于4就进行语音告警) 告警级别定义 0 信息,1 警告,2 一般严重,3 严重,4 灾难
phonecalllevel=4
#默认拨打号码(页面测试短信和电话功能需要配置此项)
defaultphone=xxxxxxxx
#故障恢复是否启用电话通知0为关闭,1为开启
phonecallresolved=0
#自动告警抑制(自动告警抑制是默认同一个告警源的告警信息只发送告警级别最高的第一条告警信息,其他消息默认屏蔽,这么做的目的是为了减少相同告警来源的消息数量,防止告警炸弹,0为关闭,1为开启)
silent=0
#是否前台输出file or console
logtype=file
#日志文件路径
logpath=logs/prometheusalertcenter.log
#转换Prometheus,graylog告警消息的时区为CST时区(如默认已经是CST时区，请勿开启)
prometheus_cst_time=0
#数据库驱动，支持sqlite3，mysql,如使用mysql，请开启db_host,db_user,db_password,db_name的注释
db_driver=sqlite3
db_host=127.0.0.1:3306
db_user=root
db_password=root
db_name=prometheusalert

#---------------------↓webhook-----------------------
#是否开启钉钉告警通道,可同时开始多个通道0为关闭,1为开启
open-dingding=1
#默认钉钉机器人地址
ddurl=https://oapi.dingtalk.com/robot/send?access_token=a1ea26b92d476b409634a74558221435b220d92b802b9373b36322586b55fb04
#是否开启 @所有人(0为关闭,1为开启)
dd_isatall=1

#是否开启微信告警通道,可同时开始多个通道0为关闭,1为开启
open-weixin=0
#默认企业微信机器人地址
wxurl=https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxx

#是否开启飞书告警通道,可同时开始多个通道0为关闭,1为开启
open-feishu=1
fsv2url=https://open.feishu.cn/open-apis/bot/v2/hook/d3784711-6fd0-4e81-a4b6-540d9e3a09f3

#---------------------↓腾讯云接口-----------------------
#是否开启腾讯云短信告警通道,可同时开始多个通道0为关闭,1为开启
open-txdx=0
#腾讯云短信接口key
TXY_DX_appkey=xxxxx
#腾讯云短信模版ID 腾讯云短信模版配置可参考 prometheus告警:{1}
TXY_DX_tpl_id=xxxxx
#腾讯云短信sdk app id
TXY_DX_sdkappid=xxxxx
#腾讯云短信签名 根据自己审核通过的签名来填写
TXY_DX_sign=腾讯云

#是否开启腾讯云电话告警通道,可同时开始多个通道0为关闭,1为开启
open-txdh=0
#腾讯云电话接口key
TXY_DH_phonecallappkey=xxxxx
#腾讯云电话模版ID
TXY_DH_phonecalltpl_id=xxxxx
#腾讯云电话sdk app id
TXY_DH_phonecallsdkappid=xxxxx

#---------------------↓华为云接口-----------------------
#是否开启华为云短信告警通道,可同时开始多个通道0为关闭,1为开启
open-hwdx=0
#华为云短信接口key
HWY_DX_APP_Key=xxxxxxxxxxxxxxxxxxxxxx
#华为云短信接口Secret
HWY_DX_APP_Secret=xxxxxxxxxxxxxxxxxxxxxx
#华为云APP接入地址(端口接口地址)
HWY_DX_APP_Url=https://rtcsms.cn-north-1.myhuaweicloud.com:10743
#华为云短信模板ID
HWY_DX_Templateid=xxxxxxxxxxxxxxxxxxxxxx
#华为云签名名称，必须是已审核通过的，与模板类型一致的签名名称,按照自己的实际签名填写
HWY_DX_Signature=华为云
#华为云签名通道号
HWY_DX_Sender=xxxxxxxxxx

#---------------------↓阿里云接口-----------------------
#是否开启阿里云短信告警通道,可同时开始多个通道0为关闭,1为开启
open-alydx=0
#阿里云短信主账号AccessKey的ID
ALY_DX_AccessKeyId=xxxxxxxxxxxxxxxxxxxxxx
#阿里云短信接口密钥
ALY_DX_AccessSecret=xxxxxxxxxxxxxxxxxxxxxx
#阿里云短信签名名称
ALY_DX_SignName=阿里云
#阿里云短信模板ID
ALY_DX_Template=xxxxxxxxxxxxxxxxxxxxxx

#是否开启阿里云电话告警通道,可同时开始多个通道0为关闭,1为开启
open-alydh=0
#阿里云电话主账号AccessKey的ID
ALY_DH_AccessKeyId=xxxxxxxxxxxxxxxxxxxxxx
#阿里云电话接口密钥
ALY_DH_AccessSecret=xxxxxxxxxxxxxxxxxxxxxx
#阿里云电话被叫显号，必须是已购买的号码
ALY_DX_CalledShowNumber=xxxxxxxxx
#阿里云电话文本转语音（TTS）模板ID
ALY_DH_TtsCode=xxxxxxxx

#---------------------↓容联云接口-----------------------
#是否开启容联云电话告警通道,可同时开始多个通道0为关闭,1为开启
RLY_DH_open-rlydh=0
#容联云基础接口地址
RLY_URL=https://app.cloopen.com:8883/2013-12-26/Accounts/
#容联云后台SID
RLY_ACCOUNT_SID=xxxxxxxxxxx
#容联云api-token
RLY_ACCOUNT_TOKEN=xxxxxxxxxx
#容联云app_id
RLY_APP_ID=xxxxxxxxxxxxx

#---------------------↓邮件配置-----------------------
#是否开启邮件
open-email=0
#邮件发件服务器地址
Email_host=smtp.qq.com
#邮件发件服务器端口
Email_port=465
#邮件帐号
Email_user=xxxxxxx@qq.com
#邮件密码
Email_password=xxxxxx
#邮件标题
Email_title=运维告警
#默认发送邮箱
Default_emails=xxxxx@qq.com,xxxxx@qq.com

#---------------------↓七陌云接口-----------------------
#是否开启七陌短信告警通道,可同时开始多个通道0为关闭,1为开启
open-7moordx=0
#七陌账户ID
7MOOR_ACCOUNT_ID=Nxxx
#七陌账户APISecret
7MOOR_ACCOUNT_APISECRET=xxx
#七陌账户短信模板编号
7MOOR_DX_TEMPLATENUM=n
#注意：七陌短信变量这里只用一个var1，在代码里写死了。
#-----------
#是否开启七陌webcall语音通知告警通道,可同时开始多个通道0为关闭,1为开启
open-7moordh=0
#请在七陌平台添加虚拟服务号、文本节点
#七陌账户webcall的虚拟服务号
7MOOR_WEBCALL_SERVICENO=xxx
# 文本节点里被替换的变量，我配置的是text。如果被替换的变量不是text，请修改此配置
7MOOR_WEBCALL_VOICE_VAR=text

#---------------------↓telegram接口-----------------------
#是否开启telegram告警通道,可同时开始多个通道0为关闭,1为开启
open-tg=0
#tg机器人token
TG_TOKEN=xxxxx
#tg消息模式 个人消息或者频道消息 0为关闭(推送给个人)，1为开启(推送给频道)
TG_MODE_CHAN=0
#tg用户ID
TG_USERID=xxxxx
#tg频道name
TG_CHANNAME=xxxxx
#tg api地址, 可以配置为代理地址
#TG_API_PROXY="https://api.telegram.org/bot%s/%s"

#---------------------↓workwechat接口-----------------------
#是否开启workwechat告警通道,可同时开始多个通道0为关闭,1为开启
open-workwechat=0
# 企业ID
WorkWechat_CropID=xxxxx
# 应用ID
WorkWechat_AgentID=xxxx
# 应用secret
WorkWechat_AgentSecret=xxxx
# 接受用户
WorkWechat_ToUser="zhangsan|lisi"
# 接受部门
WorkWechat_ToParty="ops|dev"
# 接受标签
WorkWechat_ToTag=""
# 消息类型, 暂时只支持markdown
# WorkWechat_Msgtype = "markdown"

feiyu563 commented 3 years ago

飞书的配置项名称是fsurl，新的版本已经没有fsv2url这个配置项了

python-it commented 3 years ago

飞书报警已调试成功，还有就是模板匹配问题，目前告警模板不能显示pod，namespace等信息，配置如下：报警规则：

name: kubernetes-system-apiserver rules:
- alert: KubeAPILatencyHigh annotations: message: The API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}. runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh expr: | cluster_quantile:apiserver_request_duration_seconds:histogram_quantile{job="apiserver",quantile="0.99",subresource!="log",verb!~"LIST|WATCH|WATCHLIST|PROXY|CONNECT"} > 1 for: 10m labels: severity: warning
- alert: KubeAPILatencyHigh annotations: message: The API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}. runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh expr: | cluster_quantile:apiserver_request_duration_seconds:histogram_quantile{job="apiserver",quantile="0.99",subresource!="log",verb!~"LIST|WATCH|WATCHLIST|PROXY|CONNECT"} > 4 for: 10m labels: severity: critical

报警模板： {{ $var := .externalURL}}{{ range $k,$v:=.alerts }} {{if eq $v.status "resolved"}} Prometheus恢复信息 {{$v.labels.alertname}} 告警级别：{{$v.labels.severity}} 告警namespace：{{$v.labels.namespace}} 告警pod：{{$v.labels.pod}} 告警内容：{{$v.annotations.message}} 开始时间：{{$v.startsAt}} 结束时间：{{$v.endsAt}} 故障主机IP：{{$v.labels.instance}} {{$v.annotations.description}} {{else}} Prometheus告警信息 {{$v.labels.alertname}} 告警级别：{{$v.labels.severity}} 告警namespace：{{$v.labels.namespace}} 告警pod：{{$v.labels.pod}} 告警内容：{{$v.annotations.message}} 开始时间：{{$v.startsAt}} 结束时间：{{$v.endsAt}} 故障主机IP：{{$v.labels.instance}} {{$v.annotations.description}} {{end}} {{ end }}

报警结果：

python-it commented 3 years ago

感觉是模板没有生效

feiyu563 commented 3 years ago

使用自定义模板需要配合用 /prometheusalert 自定义模板的接口才能生效

---原始邮件--- 发件人: "python-it"<notifications@github.com> 发送时间: 2021年1月13日(周三) 中午11:08 收件人: "feiyu563/PrometheusAlert"<PrometheusAlert@noreply.github.com>; 抄送: "feiyu563"<244217140@qq.com>;"State change"<state_change@noreply.github.com>; 主题: Re: [feiyu563/PrometheusAlert] 钉钉报警失效，不支持协议 (#6)

飞书报警已调试成功，还有就是模板匹配问题，目前告警模板不能显示pod，namespace等信息，配置如下：报警规则：

name: kubernetes-system-apiserver rules:
alert: KubeAPILatencyHigh annotations: message: The API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}. runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh expr: | cluster_quantile:apiserver_request_duration_seconds:histogram_quantile{job="apiserver",quantile="0.99",subresource!="log",verb!~"LIST|WATCH|WATCHLIST|PROXY|CONNECT"} > 1 for: 10m labels: severity: warning

alert: KubeAPILatencyHigh annotations: message: The API server has a 99th percentile latency of {{ $value }} seconds for {{ $labels.verb }} {{ $labels.resource }}. runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapilatencyhigh expr: | cluster_quantile:apiserver_request_duration_seconds:histogram_quantile{job="apiserver",quantile="0.99",subresource!="log",verb!~"LIST|WATCH|WATCHLIST|PROXY|CONNECT"} > 4 for: 10m labels: severity: critical

报警模板： {{ $var := .externalURL}}{{ range $k,$v:=.alerts }} {{if eq $v.status "resolved"}} Prometheus恢复信息 {{$v.labels.alertname}} 告警级别：{{$v.labels.severity}} 告警namespace：{{$v.labels.namespace}} 告警pod：{{$v.labels.pod}} 告警内容：{{$v.annotations.message}} 开始时间：{{$v.startsAt}} 结束时间：{{$v.endsAt}} 故障主机IP：{{$v.labels.instance}} {{$v.annotations.description}} {{else}} Prometheus告警信息 {{$v.labels.alertname}} 告警级别：{{$v.labels.severity}} 告警namespace：{{$v.labels.namespace}} 告警pod：{{$v.labels.pod}} 告警内容：{{$v.annotations.message}} 开始时间：{{$v.startsAt}} 结束时间：{{$v.endsAt}} 故障主机IP：{{$v.labels.instance}} {{$v.annotations.description}} {{end}} {{ end }}

报警结果：

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

python-it commented 3 years ago

我就是放在自定义模板里面的，如图：

feiyu563 commented 3 years ago

我说的是你的alertmanager的里面配置的那个URL，参考这个使用 https://github.com/feiyu563/PrometheusAlert/blob/master/doc/readme/customtpl.md

python-it commented 3 years ago

直接报警都不通了： global: resolve_timeout: 5m route: group_by: ['job'] group_wait: 10s group_interval: 5m repeat_interval: 12h receiver: webhook receivers:

name: 'webhook' webhook_configs:
- url: 'http://prometheus-alert-center.monitoring.svc.cluster.local:8080/prometheusalert' send_resolved: true

报错如下： kubectl logs prometheus-alert-center-564c49fccd-mwcnt pod -n monitoring error: container pod is not valid for pod prometheus-alert-center-564c49fccd-mwcnt

feiyu563 commented 3 years ago

仔细看文档，地址写的不对

---原始邮件--- 发件人: "python-it"<notifications@github.com> 发送时间: 2021年1月13日(周三) 中午12:01 收件人: "feiyu563/PrometheusAlert"<PrometheusAlert@noreply.github.com>; 抄送: "feiyu563"<244217140@qq.com>;"State change"<state_change@noreply.github.com>; 主题: Re: [feiyu563/PrometheusAlert] 钉钉报警失效，不支持协议 (#6)

直接报警都不通了： global: resolve_timeout: 5m route: group_by: ['job'] group_wait: 10s group_interval: 5m repeat_interval: 12h receiver: webhook receivers:

name: 'webhook' webhook_configs:
url: 'http://prometheus-alert-center.monitoring.svc.cluster.local:8080/prometheusalert' send_resolved: true

报错如下： kubectl logs prometheus-alert-center-564c49fccd-mwcnt pod -n monitoring error: container pod is not valid for pod prometheus-alert-center-564c49fccd-mwcnt

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

python-it commented 3 years ago

你文档里面就是这个地址啊

python-it commented 3 years ago

终于找对地址了

python-it commented 3 years ago

问题现象：我配置了prometheus-alert的飞书报警和非prometheus-alert项目的钉钉报警，飞书报警会延迟10分钟才会发消息至飞书机器人，配置如下： global: resolve_timeout: 5m route: group_by: ['job'] group_wait: 3s group_interval: 5s repeat_interval: 1h receiver: webhook receivers:

name: 'webhook' webhook_configs:
- url: 'http://webhook-dingtalk.monitoring.svc.cluster.local:8060/dingtalk/default-webhook-dingtalk/send' send_resolved: true
- url: 'http://prometheus-alert-center.monitoring.svc.cluster.local:8080/prometheusalert?type=fs&tpl=prometheus-fs&fsurl=https://open.feishu.cn/open-apis/bot/v2/hook/3b263649-92d3-4a9a-a4d7-xxxxxxx' send_resolved: true

feiyu563 commented 3 years ago

你试着把飞书的配置放前面

---原始邮件--- 发件人: "python-it"<notifications@github.com> 发送时间: 2021年1月14日(周四) 下午4:35 收件人: "feiyu563/PrometheusAlert"<PrometheusAlert@noreply.github.com>; 抄送: "feiyu563"<244217140@qq.com>;"State change"<state_change@noreply.github.com>; 主题: Re: [feiyu563/PrometheusAlert] 钉钉报警失效，不支持协议 (#6)

问题现象：我配置了prometheus-alert的飞书报警和非prometheus-alert项目的钉钉报警，飞书报警会延迟10分钟才会发消息至飞书机器人，配置如下： global: resolve_timeout: 5m route: group_by: ['job'] group_wait: 3s group_interval: 5s repeat_interval: 1h receiver: webhook receivers:

name: 'webhook' webhook_configs:
url: 'http://webhook-dingtalk.monitoring.svc.cluster.local:8060/dingtalk/default-webhook-dingtalk/send' send_resolved: true

url: 'http://prometheus-alert-center.monitoring.svc.cluster.local:8080/prometheusalert?type=fs&tpl=prometheus-fs&fsurl=https://open.feishu.cn/open-apis/bot/v2/hook/3b263649-92d3-4a9a-a4d7-xxxxxxx' send_resolved: true

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

python-it commented 3 years ago

确实先后顺序的问题，这个能配置同时发送吗？大佬

feiyu563 commented 3 years ago

可以，改下你的alertmanager的配置就行

---原始邮件--- 发件人: "python-it"<notifications@github.com> 发送时间: 2021年1月14日(周四) 下午4:54 收件人: "feiyu563/PrometheusAlert"<PrometheusAlert@noreply.github.com>; 抄送: "feiyu563"<244217140@qq.com>;"State change"<state_change@noreply.github.com>; 主题: Re: [feiyu563/PrometheusAlert] 钉钉报警失效，不支持协议 (#6)

确实先后顺序的问题，这个能配置同时发送吗？大佬

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

feiyu563 / PrometheusAlert

钉钉报警失效，不支持协议 #6

---------------------↓全局配置-----------------------