inCaller / prometheus_bot

Telegram bot for prometheus alerting
MIT License
395 stars 182 forks source link

Error sending message: Bad Request: can't parse entities: Expected end tag at byte offset 472 #46

Open iglov opened 4 years ago

iglov commented 4 years ago

log:

2019/10/31 11:51:38 Bot alert post: 638489237
2019/10/31 11:51:38 +------------------  A L E R T  J S O N  -------------------+
2019/10/31 11:51:38 {"alerts":[{"annotations":{"description":"Disk is almost running out of available space (\u003c 10% left)\n  VALUE = 3.102\n  LABELS: map[datacenter:fr device:/dev/mapper/san1ve-alertmanager fstype:xfs instance:st-csb2.fr:9100 job:node_exporter mountpoint:/mnt/san_volumes/alertmanager]","summary":"Out of disk space (instance st-csb2.fr:9100)"},"endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://prometheus.local:9090/graph?g0.expr=node_filesystem_free_bytes+%2F+node_filesystem_size_bytes+%2A+100+%3C+10\u0026g0.tab=1","labels":{"alertname":"OutOfDiskSpace","datacenter":"fr","device":"/dev/mapper/san1ve-alertmanager","fstype":"xfs","instance":"st-csb2.fr:9100","job":"node_exporter","mountpoint":"/mnt/san_volumes/alertmanager","severity":"warning"},"startsAt":"2019-10-31T11:49:35.277221523Z"}],"commonAnnotations":{"description":"Disk is almost running out of available space (\u003c 10% left)\n  VALUE = 3.102\n  LABELS: map[datacenter:fr device:/dev/mapper/san1ve-alertmanager fstype:xfs instance:st-csb2.fr:9100 job:node_exporter mountpoint:/mnt/san_volumes/alertmanager]","summary":"Out of disk space (instance st-csb2.fr:9100)"},"commonLabels":{"alertname":"OutOfDiskSpace","datacenter":"fr","device":"/dev/mapper/san1ve-alertmanager","fstype":"xfs","instance":"st-csb2.fr:9100","job":"node_exporter","mountpoint":"/mnt/san_volumes/alertmanager","severity":"warning"},"externalURL":"http://alertmanager.fr.local:9093","groupKey":0,"groupLabels":{"alertname":"OutOfDiskSpace","instance":"st-csb2.fr:9100"},"receiver":"adm","status":"firing","version":0}
2019/10/31 11:51:38 +-----------------------------------------------------------+
2019/10/31 11:51:38 +---------------  F I N A L   M E S S A G E  ---------------+
2019/10/31 11:51:38 <a href='http://alertmanager.fr.local:9093/#/alerts?receiver=adm'>[FIRING:1]</a>
grouped by: alertname=<code>OutOfDiskSpace</code>, instance=<code>st-csb2.fr:9100</code>
labels: datacenter=<code>fr</code>, device=<code>/dev/mapper/san1ve-alertmanager</code>, fstype=<code>xfs</code>, job=<code>node_exporter</code>, mountpoint=<code>/mnt/san_volumes/alertmanager</code>, severity=<code>warning</code>
description: <code>Disk is almost running out of available space (< 10% left)
VALUE = 3.102
LABELS: map[datacenter:fr device:/dev/mapper/san1ve-alertmanager fstype:xfs instance:st-csb2.fr:9100 job:node_exporter mountpoint:/mnt/san_volumes/alertmanager]</code>
summary: <code>Out of disk space (instance st-csb2.fr:9100)</code>
<a href='http://prometheus.local:9090/graph?g0.expr=node_filesystem_free_bytes+%2F+node_filesystem_size_bytes+%2A+100+%3C+10&g0.tab=1'>st-csb2.fr[node_exporter]</a>
2019/10/31 11:51:38 +-----------------------------------------------------------+
2019/10/31 11:51:38 Error sending message: Bad Request: can't parse entities: Expected end tag at byte offset 472
[GIN] 2019/10/31 - 11:51:38 | 503 |   66.474775ms | 192.168.100.128 | POST     /alert/638489237

config.yaml :

telegram_token: "5327342:GVJsadjgvwequJHSAGh"
time_zone: "Europe/Sofia"
split_token: "|"

alertmanager config:

  - alert: OutOfDiskSpace
    expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Out of disk space (instance {{ $labels.instance }})"
      description: "Disk is almost running out of available space (< 10% left)\n  VALUE = {{ $value | humanize }}\n  LABELS: {{ $labels }}"
#      description: "Disk is almost full (< 10% left)\n {{ $labels.instance_short }}\n {{ $labels.mountpoint }}\n VALUE = {{ printf \"node_filesystem_avail_bytes{mountpoint='%s'}\" .Labels.mountpoint | query | first | value | humanize1024 }}"

The problem with description - (< 10% left) , special symbols can not be parsed.

P.S. I am not a programmer and i dunno how to fix it, so i can't make a PR, sry.

antonrubtsov commented 4 years ago

may be you should use &lt; instead of \u003c

iglov commented 4 years ago

hi, @antonrubtsov ! I should use human readable format in my alertmanager config like this description: "Disk is almost running out of available space (< 10% left) and i believe it should be parsed in code.

strangeman commented 4 years ago

Try our fork, we fixed HTML sanitization issues (https://github.com/UnitedTraders/prometheus_bot)

hryamzik commented 4 years ago

@strangeman where's the PR? ;-)

strangeman commented 4 years ago

Will do, my apologies :)

On Thu, 31 Oct 2019, 22:46 Roman Belyakovsky, notifications@github.com wrote:

@strangeman https://github.com/strangeman where's the PR? ;-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/inCaller/prometheus_bot/issues/46?email_source=notifications&email_token=AAIIROHXJK6LRG3XIT2YKE3QRMY2LA5CNFSM4JHJDCFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECZA2CI#issuecomment-548539657, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIIROHQNFORUH37ASXAXILQRMY2LANCNFSM4JHJDCFA .

iglov commented 4 years ago

Try our fork, we fixed HTML sanitization issues (https://github.com/UnitedTraders/prometheus_bot)

Same problem here...