search

benjojo / alertmanager-discord

Take your alertmanager alerts, into discord
Apache License 2.0
177 stars 78 forks source link

Error on alert - cannot unmarshal array into Go value of type main.alertManOut #18

Closed sfoxdev closed 3 years ago

sfoxdev commented 3 years ago

Using docker image benjojo/alertmanager-discord:latest Runs with Prometheus 2.24.0 and Grafana 7.3.6.

On alert getting such error:

2021/02/17 16:03:29 Listening on: 0.0.0.0:9094
2021/02/17 16:03:52 http: panic serving 10.0.8.4:57922: json: cannot unmarshal array into Go value of type main.alertManOut
goroutine 6 [running]:
net/http.(*conn).serve.func1(0xc00006ebe0)
        /usr/local/go/src/net/http/server.go:1801 +0x147
panic(0x6dd0c0, 0xc00004c230)
        /usr/local/go/src/runtime/panic.go:975 +0x47a
main.main.func1(0x79a560, 0xc0000b80e0, 0xc0000ac100)
        /go/src/mypackage/myapp/main.go:110 +0xd54
net/http.HandlerFunc.ServeHTTP(0xc000040c90, 0x79a560, 0xc0000b80e0, 0xc0000ac100)
        /usr/local/go/src/net/http/server.go:2042 +0x44
net/http.serverHandler.ServeHTTP(0xc0000b8000, 0x79a560, 0xc0000b80e0, 0xc0000ac100)
        /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc00006ebe0, 0x79ab60, 0xc000010500)
        /usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2969 +0x36c
2021/02/17 16:03:52 http: panic serving 10.0.8.4:57924: json: cannot unmarshal array into Go value of type main.alertManOut
benjojo commented 3 years ago

What is your configuration for alert manager or anything else sending things to that container?

On Wed, 24 Feb 2021, 09:49 SFoxDev, notifications@github.com wrote:

Using docker image benjojo/alertmanager-discord:latest Runs with Prometheus 2.24.0 and Grafana 7.3.6.

On alert getting such error:

2021/02/17 16:03:29 Listening on: 0.0.0.0:9094 2021/02/17 16:03:52 http: panic serving 10.0.8.4:57922: json: cannot unmarshal array into Go value of type main.alertManOut goroutine 6 [running]: net/http.(conn).serve.func1(0xc00006ebe0) /usr/local/go/src/net/http/server.go:1801 +0x147 panic(0x6dd0c0, 0xc00004c230) /usr/local/go/src/runtime/panic.go:975 +0x47a main.main.func1(0x79a560, 0xc0000b80e0, 0xc0000ac100) /go/src/mypackage/myapp/main.go:110 +0xd54 net/http.HandlerFunc.ServeHTTP(0xc000040c90, 0x79a560, 0xc0000b80e0, 0xc0000ac100) /usr/local/go/src/net/http/server.go:2042 +0x44 net/http.serverHandler.ServeHTTP(0xc0000b8000, 0x79a560, 0xc0000b80e0, 0xc0000ac100) /usr/local/go/src/net/http/server.go:2843 +0xa3 net/http.(conn).serve(0xc00006ebe0, 0x79ab60, 0xc000010500) /usr/local/go/src/net/http/server.go:1925 +0x8ad created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2969 +0x36c 2021/02/17 16:03:52 http: panic serving 10.0.8.4:57924: json: cannot unmarshal array into Go value of type main.alertManOut

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/benjojo/alertmanager-discord/issues/18, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALPK4W26MBV477ZYRRTDPLTATDRNANCNFSM4YEFYAYQ .

sfoxdev commented 3 years ago

Here is config file

global:
  resolve_timeout: 1m

route:
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 30m
  receiver: 'discord'

receivers:
  - name: 'discord'
    webhook_configs:
    - url: 'http://localhost:9094'
benjojo commented 3 years ago

Do you have health/readiness checks enabled for the container by any chance? It might be having issues with those.

sfoxdev commented 3 years ago

No, I tried to run using simple docker-compose and Docker Swarm, result the same.

benjojo commented 3 years ago

@sfoxdev Please pull :latest and reproduce the issue, then report back the output?

benjojo commented 3 years ago

@sfoxdev have you had a moment to try this?

sfoxdev commented 3 years ago

@benjojo I updated container to latest and on alert in logs I getting such message, but doesn't receive alert in Discord.

2021/02/27 21:17:37 alertmanager-discord:9094 - [POST]
2021/02/27 21:17:37 Failed to unpack inbound alert request - [{"labels":{"alertname":"messaging_srv","container_label_com_docker_swarm_service_name":"messaging_srv","monitor":"exporter-metrics","server":"MESSAGING","severity":"critical"},"annotations":{"description":"Messaging SRV container is down for more than 10 seconds.","summary":"MESSAGING: API down"},"startsAt":"2021-02-27T21:02:22.036737213Z","endsAt":"2021-02-27T21:02:37.036737213Z","generatorURL":"http://65f20cc80261:9090/graph?g0.expr=absent%28container_memory_usage_bytes%7Bcontainer_label_com_docker_swarm_service_name%3D%22messaging_srv%22%2Cserver%3D%22MESSAGING%22%7D%29\u0026g0.tab=1"}]
benjojo commented 3 years ago

Interesting, Your alertmanager is ending them as an array of webhooks rather than a singular, Must be a new thing, let me investigate

benjojo commented 3 years ago

Nope. You have misconfigured this, You need alertmanager as well as alertmanager-discord.

The dataflow should be:

Prometheus -> Alertmanager -> Alertmanager-discord

You have:

Prometheus -> Alertmanager-discord

I'll build in protection for this now, since it keeps happening

benjojo commented 3 years ago

When this happens now, This following will appear: image