Alertmanager example

[skluthe]

Is there any way you could provide an example configuration for alert manager? The follow is what I'm using without luck:

route:
  receiver: 'notify'
receivers:
  - name: 'notify'
    webhook_configs:
      - url: 'http://notify-bridge:8080/gotify_webhook'

[skluthe]

Still having trouble getting this to work. Here are my logs: gotify_bridge:

notify-bridge    | Starting debug server on http://0.0.0.0:8080/gotify_webhook translating to http://notify:80/message ...
notify-bridge    | WARNING: /message not at the end of the gotify_endpoint parameter (http://notify:80). Automatically appending it.
notify-bridge    | New gotify_endpoint: http://notify:80/message
notify-bridge    | 2021/02/25 20:47:31 bridge: Recieved request: &{Method:POST URL:/gotify_webhook Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[1267] Content-Type:[application/json] User-Agent:[Alertmanager/0.21.0]] Body:0xc000072500 GetBody:<nil> ContentLength:1267 TransferEncoding:[] Close:false Host:notify-bridge:8080 Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr:172.20.0.8:49256 RequestURI:/gotify_webhook TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000072540}
notify-bridge    | 2021/02/25 20:47:31 bridge: Headers:
notify-bridge    | 2021/02/25 20:47:31 bridge:  content-length: 1267
notify-bridge    | 2021/02/25 20:47:31 bridge:  content-type: application/json
notify-bridge    | 2021/02/25 20:47:31 bridge:  user-agent: Alertmanager/0.21.0
notify-bridge    | 2021/02/25 20:47:31 bridge: BODY: {"receiver":"notify","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"TEST ALERT FROM PROMETHEUS PLEASE ACKNOWLEDGE","branch":"HEAD","goversion":"go1.15.8","instance":"localhost:9090","job":"prometheus","monitor":"traefik","revision":"a6be548dbc17780d562a39c0e4bd0bd4c00ad6e2","severity":"warning","version":"2.25.0"},"annotations":{"action":"TESTING PLEASE ACKNOWLEDGE, NO FURTHER ACTION REQUIRED ONLY A TEST","description":"TEST ALERT FROM localhost:9090"},"startsAt":"2021-02-25T18:21:16.646300974Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://3da5b2e699c3:9090/graph?g0.expr=prometheus_build_info%7Binstance%3D%22localhost%3A9090%22%7D+%3D%3D+1\u0026g0.tab=1","fingerprint":"e18369a3f432dd69"}],"groupLabels":{},"commonLabels":{"alertname":"TEST ALERT FROM PROMETHEUS PLEASE ACKNOWLEDGE","branch":"HEAD","goversion":"go1.15.8","instance":"localhost:9090","job":"prometheus","monitor":"traefik","revision":"a6be548dbc17780d562a39c0e4bd0bd4c00ad6e2","severity":"warning","version":"2.25.0"},"commonAnnotations":{"action":"TESTING PLEASE ACKNOWLEDGE, NO FURTHER ACTION REQUIRED ONLY A TEST","description":"TEST ALERT FROM localhost:9090"},"externalURL":"http://73374d8cc407:9093","version":"4","groupKey":"{}:{}","truncatedAlerts":0}
notify-bridge    |
notify-bridge    | 2021/02/25 20:47:31 bridge: data sent - unmarshalling from JSON: {"receiver":"notify","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"TEST ALERT FROM PROMETHEUS PLEASE ACKNOWLEDGE","branch":"HEAD","goversion":"go1.15.8","instance":"localhost:9090","job":"prometheus","monitor":"traefik","revision":"a6be548dbc17780d562a39c0e4bd0bd4c00ad6e2","severity":"warning","version":"2.25.0"},"annotations":{"action":"TESTING PLEASE ACKNOWLEDGE, NO FURTHER ACTION REQUIRED ONLY A TEST","description":"TEST ALERT FROM localhost:9090"},"startsAt":"2021-02-25T18:21:16.646300974Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://3da5b2e699c3:9090/graph?g0.expr=prometheus_build_info%7Binstance%3D%22localhost%3A9090%22%7D+%3D%3D+1\u0026g0.tab=1","fingerprint":"e18369a3f432dd69"}],"groupLabels":{},"commonLabels":{"alertname":"TEST ALERT FROM PROMETHEUS PLEASE ACKNOWLEDGE","branch":"HEAD","goversion":"go1.15.8","instance":"localhost:9090","job":"prometheus","monitor":"traefik","revision":"a6be548dbc17780d562a39c0e4bd0bd4c00ad6e2","severity":"warning","version":"2.25.0"},"commonAnnotations":{"action":"TESTING PLEASE ACKNOWLEDGE, NO FURTHER ACTION REQUIRED ONLY A TEST","description":"TEST ALERT FROM localhost:9090"},"externalURL":"http://73374d8cc407:9093","version":"4","groupKey":"{}:{}","truncatedAlerts":0}
notify-bridge    |
notify-bridge    | 2021/02/25 20:47:31 Detected 1 alerts
notify-bridge    | 2021/02/25 20:47:31   title annotation ("Prometheus Alert") missing
notify-bridge    | 2021/02/25 20:47:31   message annotation (summary) missing
notify-bridge    | 2021/02/25 20:47:31   priority annotation (priority) missing - falling back to default (5)

alertmanager:

alertmanager     | level=error ts=2021-02-25T20:47:31.655Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="notify/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: http://notify-bridge:8080/gotify_webhook"

No message appears in gotify.

[skluthe]

Here is my docker-compose:

  notify:
    image: gotify/server:latest
    container_name: notify
    volumes:
      - notify_data:/app/data
    networks:
      - web
    labels:
      - org.label-schema.group=monitoring

  notify-bridge:
    container_name: notify-bridge
    image: nathaneaston/alertmanager_gotify_bridge-docker
    networks:
      - web
    ports:
      - 8080:8080
    environment:
      - GOTIFY_TOKEN=<actual token here>
      - ENDPOINT=http://notify:80
      - TITLE_ANNOTATION="Prometheus Alert"
      - MESSAGE_ANNOTATION="Alert from prometheus incoming"

[DRuggeri]

Hello, @skluthe - sure. Here's the relevant alertmanager config I have set up. It's pretty unremarkable:

    route:
      group_by: ['alertname', 'priority']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: alerter

    receivers:
    - name: alerter
      email_configs:
      - to: 'my@emailaddress.foo'
        send_resolved: false
      webhook_configs:
      - url: http://127.0.0.1:81/gotify_webhook
        send_resolved: false

The only thing of note here is that I'm running the bridge on port 81 because I already have a service on 80.

It looks like the real problem is the 400 reported by alertmanager. There seems to be something about the POST the bridge makes that alertmanager doesn't like. I'm not sure if there's a more robust debug log that can be enabled on the alertmanager side to see if the problem could be more clear? I took a look at the data POSTed from the bridge thanks to the debug log you provided and can confirm that it's at least clean JSON and has the minimum expected parameters.

[DRuggeri]

Actually - on second review... you should be seeing a message Required fields found. Dispatching to gotify..., but that's missing. I think the bridge is not attempting to dispatch because of the missing fields noted. Makes sense - the title and message fields are required. I think you just need to set --title_annotation=action and --message_annotation=description based on the content of the prometheus alert.

[skluthe]

That did it. Thanks!