URL alermanager webhook not loaded from alertmanager-yml

[GeluConstantin]

**Configured alertmanager.yml

route: receiver: 'email' group_by: ['alertname'] group_wait: 30s group_interval: 5m repeat_interval: 3h routes:

• match: service: fme-server receiver: 'custom-email'

• match: service: fme-server receiver: 'webhook2'

  - match:

  service: fme-server

  receiver: 'email'

  - match:

  service: node-exporter

  receiver: 'email'

receivers:

• name: 'webhook2' webhook_configs:
  • send_resolved: true http_config: {} url: 'http://localhost:8031/api/monitoring-alerts' max_alerts: 5

**

Correct configured Webhook, like in Grafa UI

What did you see instead? Under which circumstances? All other parameters are loaded but not the URL

**

Environment

• System information:

  insert output of uname -srm here

• Alertmanager version:

  insert output of alertmanager --version here (repeat for each alertmanager version in your cluster, if relevant to the issue)

• Prometheus version:

  insert output of prometheus --version here (repeat for each prometheus version in your cluster, if relevant to the issue)

• Alertmanager configuration file:

  alertmanager, version 0.24.0 (branch: HEAD, revision: f484b17fa3c583ed1b2c8bbcec20ba1db2aa5f11)
  build user:       root@265f14f5c6fc
  build date:       20220325-09:31:33
  go version:       go1.17.8
  platform:         linux/amd64

• Prometheus configuration file:

  prometheus, version 2.47.0 (branch: HEAD, revision: efa34a5840661c29c2e362efa76bc3a70dccb335)
  build user:       root@409eb5e6b30c
  build date:       20230906-10:20:07
  go version:       go1.21.0
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

• Logs:

  nothing logged by Alermanager, even in debug mode

[slvr32]

I'm not sure if this is the same issue, but with alertmanager 0.26.0, the following in alertmanager yaml

receivers:

• name: ms_teams_receiver webhook_configs:

  • send_resolved: true http_config: {} url: http://localhost:2000/alertmanager

  produces the following on the alertmanager status page...

  receivers:

• name: ms_teams_receiver webhook_configs:
  • send_resolved: true http_config: follow_redirects: true enable_http2: true url: url_file: "" max_alerts: 0

whereas the same yaml with a much older alertmanager ( 0.19.0 ), shows the expected configuration on the alertmanager status page

receivers:

• name: ms_teams_receiver webhook_configs:
  • send_resolved: true http_config: {} url: http://localhost:2000/alertmanager

at a glance, it looks like the http_config: {} is being ignored, and the url value is being turned into a 'secret' value?

[slvr32]

I noticed that masking the url value with was added in https://github.com/prometheus/alertmanager/pull/3228

The interesting other details is that attempting a curl against one of the 'secret' webhook URLs results in 404, whereas the older ( 0.19.0 ) alertmanager reports that it expects a POST to a particular webhook URL

$ curl -v localhost:2000/alertmanager

• Trying 127.0.0.1...
• TCP_NODELAY set
• Connected to localhost (127.0.0.1) port 2000 (#0)

  GET /alertmanager HTTP/1.1 Host: localhost:2000 User-Agent: curl/7.61.1 Accept: /

  < HTTP/1.1 404 Not Found < Content-Type: application/json; charset=UTF-8 < Date: Wed, 10 Jan 2024 21:23:38 GMT < Content-Length: 24 < {"message":"Not Found"}

• Connection #0 to host localhost left intact

vs behavior with alertmanager 0.19.0

$ curl -v localhost:2000/alertmanager

• About to connect() to localhost port 2000 (#0)
• Trying 127.0.0.1... connected
• Connected to localhost (127.0.0.1) port 2000 (#0)

  GET /alertmanager HTTP/1.1 User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 Host: localhost:2000 Accept: /

  < HTTP/1.1 405 Method Not Allowed < Content-Type: text/plain; charset=utf-8 < X-Content-Type-Options: nosniff < Date: Wed, 10 Jan 2024 21:32:21 GMT < Content-Length: 69 < Invalid request method: GET, this handler only accepts POST requests

• Connection #0 to host localhost left intact
• Closing connection #0

[slvr32]

Oops, disregard the tcp 2000 curl discussion, since that's related to behavior of the actual prometheus-msteams receiver binary