Alertmanager sent extra line in JSON payload to webhook endpoint

[willz9683]

What did you do? Alertmanager integration with external webhook API interface. Use postman to analysis alertmanager POST request.

What did you expect to see? alert JSON message in one string of POST request payload

What did you see instead? Under which circumstances? alertmanager send extra empty line (newline?) as part of the payload. Payload is considered as invalid JSON format.

Environment

• System information:

  # uname -srm
  Linux 4.18.0-193.el8.x86_64 x86_64

• Alertmanager version:

  # alertmanager --version
  alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d)
  build user:       root@dee35927357f
  build date:       20200617-08:54:02
  go version:       go1.14.4

• Prometheus version:

  # prometheus --version
  prometheus, version 2.22.1 (branch: HEAD, revision: 00f16d1ac3a4c94561e5133b821d8e4d9ef78ec2)
  build user:       root@516b109b1732
  build date:       20201105-14:02:25
  go version:       go1.15.3
  platform:         linux/amd64

• Alertmanager configuration file:

  
  global:
  resolve_timeout: 5s
  templates: 

• '/etc/prometheus/alertmanager_templates/*.tmpl'

route: group_by: ['alertname', 'cluster', 'service'] group_wait: 5s group_interval: 5m repeat_interval: 3h receiver: "snmp_webhook" routes:

• receiver: "snmp_webhook"

inhibit_rules:

• source_match: severity: 'critical' target_match: severity: 'warning' equal: ['alertname', 'cluster', 'service']

receivers:

• name: 'snmp_webhook' webhook_configs:
  • url: 'http://localhost:9099'
  • url: 'http://
  • url: 'http://localhost:5555'

* Logs:

level=error ts=2020-11-09T15:10:22.946Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected status code 400: http://"

postman analysis snapshots are attached.

![alertmanager_post_payload](https://user-images.githubusercontent.com/70406217/98559429-9a95ee80-2274-11eb-973c-d7ae67e7d82d.png)

[roidelapluie]

I think that this is the receiver's fault. We use standard go tools for this, and they are accepted by the majority of receivers. What is the receiver?

[willz9683]

The receiver is DMaaP Message Router

[simonpasquier]

Reading https://tools.ietf.org/html/rfc7159 I see nothing wrong here. As Julien says, we use the standard Go library to produce the JSON payload and it would be surprising if it were generating incompatible payload.