benjojo / alertmanager-discord

Take your alertmanager alerts, into discord
Apache License 2.0
177 stars 78 forks source link

Discord messages are empty #26

Closed Lusitaniae closed 2 years ago

Lusitaniae commented 2 years ago

Did discord have some update that broke formatting?

I set this up and i have a bunch of alerts that only show summary, e.g.

=== Network errors over x ===

=== Disk is full ===

For others, I can see the notification in my desktop (something something error on server x)

but the message shows empty

image

benjojo commented 2 years ago

Did it used to work? Or is this a new install?

Lusitaniae commented 2 years ago

First time!

Now I'm thinking it could be because I'm using VictoriaMetrics VMAlert (instead of Alertmanager).

They're similar, but could have a few differences in which case no issue at all, i'm checking now

benjojo commented 2 years ago

Ah! Critical detail left out ;)

If possible, can you tcpdump (or obtain otherwise) what VictoriaMetrics VMAlert sends to the alertmanager-discord port?

Lusitaniae commented 2 years ago

new setup for me, that's why didn't come up at first

I'll update the issue if i find something

benjojo commented 2 years ago

If you could give me the payload that VictoriaMetrics VMAlert sends to the alertmanager-discord port I could help you debug this faster :)

Lusitaniae commented 2 years ago

Sorry for the delay,

Very appreciated for looking into this, more than needed or expected really :)

I could continue to investigate on my own and do PR if required or update the issue with findings

{"receiver":"discord-notifications","status":"firing","alerts":[{"status":"firing","labels":{"alertgroup":"group","alertname":"TXTransactionRetryCount","cluster":"name","host_id":"c3KHnL2CKGNb","instance":"11.11.11.11","job":"job","severity":"warning"},"annotations":{"description":"The transaction retries on  is \u003e 50 VALUE = 366 LABELS = map[alertgroup:group cluster:name host_id:c3KHnL2CKGNb instance:11.11.11.11 job:job severity:warning]","summary":"The transaction retry count on 11.11.11.11 is \u003e 50"},"startsAt":"2021-12-01T14:59:47.31714679Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://mon-vmagent1:8880/api/v1/11111111111/11111111111/status","fingerprint":"7af988f4f8f41fc2"},{"status":"firing","labels":{"alertgroup":"group","alertname":"TXTransactionRetryCount","cluster":"name","host_id":"c3KHnL2CKGNb","instance":"12.12.12.12","job":"job","severity":"warning"},"annotations":{"description":"The transaction retries on  is \u003e 50 VALUE = 54 LABELS = map[alertgroup:group cluster:name host_id:c3KHnL2CKGNb instance:12.12.12.12 job:job severity:warning]","summary":"The transaction retry count on 12.12.12.12 is \u003e 50"},"startsAt":"2021-12-01T14:50:47.315453252Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://mon-vmagent1:8880/api/v1/11111111111/11111111111/status","fingerprint":"393db6f774944035"},{"status":"firing","labels":{"alertgroup":"group","alertname":"TXTransactionRetryCount","cluster":"name","host_id":"c3KHnL2CKGNb","instance":"13.13.13.13","job":"job","severity":"warning"},"annotations":{"description":"The transaction retries on  is \u003e 50 VALUE = 101 LABELS = map[alertgroup:group cluster:name host_id:c3KHnL2CKGNb instance:13.13.13.13 job:job severity:warning]","summary":"The transaction retry count on 13.13.13.13 is \u003e 50"},"startsAt":"2021-12-01T13:53:47.313901255Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":"http://mon-vmagent1:8880/api/v1/11111111111/11111111111/status","fingerprint":"4025294f01958ee9"}],"groupLabels":{"alertname":"TXTransactionRetryCount","cluster":"name"},"commonLabels":{"alertgroup":"group","alertname":"TXTransactionRetryCount","cluster":"name","job":"job","severity":"warning"},"commonAnnotations":{},"externalURL":"http://localhost:9093","version":"4","groupKey":"{}/{severity=\"warning\"}:{alertname=\"TXTransactionRetryCount\", cluster=\"name\"}","truncatedAlerts":0}

I replaced some values for "privacy"

benjojo commented 2 years ago

That's interesting, this fires on my system no worries.

image

benjojo commented 2 years ago

I just rebuild this from source, and replayed that payload. and it worked as well.

Are there any error messages from the program on your side? I can't see why this can't work

Lusitaniae commented 2 years ago

Nothing in the logs

image

I'm trying local build too (printing the json sent to discord), same result

./alertmanager-discord -webhook.url https://discord.com/api/webhooks/token/token
2021/12/01 23:37:22 Listening on: 127.0.0.1:9094
2021/12/01 23:37:24 127.0.0.1:9094 - [POST] 
{"content":"","embeds":[{"title":"[FIRING:3] TXTransactionRetryCount","type":"rich","description":"","color":10038562,"fields":[{"name":"[FIRING]: TXTransactionRetryCount on 145.40.93.223","value":"The transaction retri

I'm new to Discord too, could this require paid feature hah?

Lusitaniae commented 2 years ago

Just found it

Some personal settings somehow affect webhook messages uhhhh

User Settings -> Text & Images -> Show website preview info from links pasted in chat

For some odd reason, this setting also applies to message embeds by bots too :/

Thank you for looking into it!

benjojo commented 2 years ago

Cool, Glad you fixed it