telegram: Bad Request: can't parse entities: Unsupported start tag \"\" at byte offset 218 (400)

[krax1337]

Hello! My receiver is telegram. I think alermanager doesn't escape text properly. Parse mode is HTML.

Alertmanager config:

receivers:
  - name: "telegram"
    telegram_configs:
      # TODO set token
      - bot_token: 
        # TODO set chatid
        chat_id: 
        api_url: https://api.telegram.org
        parse_mode: HTML

Alert that causes this problem:

      - alert: HostOutOfMemory
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: Host out of memory (instance {{ $labels.instance }})
          description: "Node memory is filling up (< 10% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

Full log text: ts=2022-09-07T10:18:07.778Z caller=notify.go:732 level=warn component=dispatcher receiver=telegram integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telegram: Bad Request: can't parse entities: Unsupported start tag \"\" at byte offset 218 (400)"

https://github.com/prometheus/alertmanager/issues/2960 - I think that is the same, but I didn't find a solution.

[Dem0n3D]

Same problem. As a workaround, remove special HTML chars from description (change < 10% left to < 10% left).

[mbugert]

Same problem here using prom/alertmanager:v0.24.0, but with the default parse mode (which should be MarkdownV2):

ts=2022-11-08T06:38:22.684Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="telegram/telegram[0]: notify retry canceled after 16 attempts: telegram: Bad Request: can't parse entities: Character '-' is reserved and must be escaped with the preceding '\\' (400)"

[rmn-lux]

Also have the same problem, alertmanager v0.24, parse mode html

[rosberen]

Also have the same problem, alertmanager v0.24

[simonpasquier]

I suppose that we need to notify.TmplHTML() instead of notify.TmplText() when parse_mode is HTML.

[xluffy]

Same issue with default parse mode MarkdownV2

[jafnhaar]

Same issue here with MarkdownV2 and HTML parse mode.

[timur-enikeev]

Is there any workaround?

My config is:

- name: telega
  telegram_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
    api_url: https://api.telegram.org
    bot_token: <secret>
    chat_id: <secret>
    message: '*Details:* {{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
    parse_mode: MarkdownV2

And I receive:

ts=2022-12-16T16:23:53.133Z caller=notify.go:732 level=warn component=dispatcher receiver=telega integration=telegram[0] msg="Notify attempt failed, will retry later" attempts=1 err="telegram: Bad Request: can't parse entities: Character '-' is reserved and must be escaped with the preceding '\\' (400)"

Alertmenager is installed from https://hub.docker.com/r/prom/alertmanager/ I tried version v0.24.0 and main build few days ago.

[jafnhaar]

I switched to HTML parse mode and removed all special characters from alerts description (i.e. switched "<" with "<") and it worked like a charm

[timur-enikeev]

@jafnhaar

could you provide an example?

[Dem0n3D]

@jafnhaar

could you provide an example?

You need to remove special chars from your rules, like this:

   - alert: HostOutOfMemory
      expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: Host out of memory (instance {{ $labels.instance }})
        description: "Node memory is filling up (&lt; 10% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

[timur-enikeev]

@Dem0n3D Should I rewrite every alert? It is not possible. Is any workaround to fix the bug without rewriting every alert?

[jafnhaar]

I don't think there is a workaround without changing rules description.