search

benjojo / alertmanager-discord

Take your alertmanager alerts, into discord
Apache License 2.0
177 stars 78 forks source link

Failed to unpack inbound alert request #24

Open acmodeu opened 3 years ago

acmodeu commented 3 years ago

I want to send some alerts from external exporter via kubernetes prometheus stack to discord. I see the alerts in the alertmanager but they don't appear in discord channel and there are a lot of similar errors in alertmanager-discord logs.

2021/08/12 19:46:39 10.200.129.205:9094 - [GET] 
2021/08/12 19:46:39 Failed to unpack inbound alert request -
benjojo commented 3 years ago

Are you sure the alert manager is actually sending the alert webhooks to this software?

Those logs you provided look like they are just health checks, presumably from k8s On Thu, 12 Aug 2021, 20:50 acmodeu, @.***> wrote:

I want to send some alerts from external exporter via kubernetes prometheus stack to discord. I see the alerts in the alertmanager but they don't appear in discord channel and there are a lot of similar errors in alertmanager-discord logs.

2021/08/12 19:46:39 10.200.129.205:9094 - [GET] 2021/08/12 19:46:39 Failed to unpack inbound alert request -

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/benjojo/alertmanager-discord/issues/24, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALPK4WM2MEKBMHAEGEDJHDT4QQX5ANCNFSM5CCARETA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

acmodeu commented 3 years ago

Yes. I'm sure, I set up discord as default receiver for all alerts but received all other kubernetes alerts except those external ones. I switched to telegram bot and received them right away.

benjojo commented 3 years ago

So, alertmanager webhooks come in as POST's so you should look for something like

2021/08/12 19:46:39 10.200.129.205:9094 - [POST]

in your logs, if you can't find them, webhooks are not directly landing on alertmanager-discord.

h4wkmoon commented 3 years ago

Hi,

When using into k8s, you may want to use readiness and liveness probes. They usually make GET requests to the app, to see if the process is alive and fit for duty. If something other than 2xx is returned, kubernetes will kill the container, and spawn another one.