prometheus / alertmanager

Prometheus Alertmanager
https://prometheus.io
Apache License 2.0
6.64k stars 2.15k forks source link

I can get the attribute 'updateAt from Alartmanager API api/v2/alerts but no this from webhook #2142

Open ghost opened 4 years ago

ghost commented 4 years ago

question I can get the attribute 'updateAt from Alartmanager API api/v2/alerts but no this from webhook。 Raw data as below: json.loads(request.data) Clould you please share some ideas?

VERSION V0.20.0

json.loads(request.data)

< class 'dict' >: {
    'receiver': 'webhook',
    'status': 'firing',
    'alerts': [{
        'status': 'firing',
        'labels': {
            'Node': 'instance',
            'alertname': '节点状态',
            'instance': 'localhost:9100',
            'job': 'node-export',
            'severity': 'Critical'
        },
        'annotations': {
            'description': '当前服务器可能已经宕机,请及时排查问题',
            'summary': '服务器故障',
            'value': '0'
        },
        'startsAt': '2019-12-17T11:56:20.873303345Z',
        'endsAt': '0001-01-01T00:00:00Z',
        'generatorURL': 'http://prometheus:9090/graph?g0.expr=up%7Bjob%3D%22node-export%22%7D+%3D%3D+0&g0.tab=1',
        'fingerprint': '24eccb8d84d71e24'
    }],
    'groupLabels': {},
    'commonLabels': {
        'Node': 'instance',
        'alertname': '节点状态',
        'instance': 'localhost:9100',
        'job': 'node-export',
        'severity': 'Critical'
    },
    'commonAnnotations': {
        'description': '当前服务器可能已经宕机,请及时排查问题',
        'summary': '服务器故障',
        'value': '0'
    },
    'externalURL': 'http://alertmanager:9093',
    'version': '4',
    'groupKey': '{}:{}'
}

GET http://localhost:9093/api/v2/alerts

"annotations":{
"description": "当前服务器可能已经宕机,请及时排查问题",
"summary": "服务器故障",
"value": "0"
},
"endsAt": "2019-12-17T12:03:05.873Z",
"fingerprint": "24eccb8d84d71e24",
"receivers":[
{
"name": "webhook"
}
],
"startsAt": "2019-12-17T11:56:20.873Z",
"status":{
"inhibitedBy":[],
"silencedBy":[],
"state": "active"
},
"updatedAt": "2019-12-17T12:00:05.875Z",
simonpasquier commented 4 years ago

There's no obvious reason why the receiver of the notification should care when the alert got updated. This is why the webhook payload doesn't include this information.

ghost commented 4 years ago

There's no obvious reason why the receiver of the notification should care when the alert got updated. This is why the webhook payload doesn't include this information.

no! no ! no !

I think it's in the API, so it should be in webhook. Using webhook requires tracking alarm times. startsAt endAt updateAt. If I don't handle the alarm and set the repeated alarm to 1 hour, then the next alarm time will still be startsAt, which will lead to misunderstanding

The time in the figure below comes from startsAt. The startsAt time is the same, but the alarm value is different, and the sending time is different

企业微信截图_1576734768743

simonpasquier commented 4 years ago

As the update timestamp is per Alertmanager, it wouldn't be consistent in case of Alertmanager cluster (a notification from Alertmanager A wouldn't have the same updatedAt value than Alertmanager B or C).