search

graylog-labs / graylog-plugin-slack

Graylog alarm callback for Slack
https://www.graylog.org
Apache License 2.0
117 stars 52 forks source link

I don't see the actual message which caused an alert in Slack #56

Closed derFunk closed 6 years ago

derFunk commented 7 years ago

This is my alert condition: Configuration: Alert is triggered when the field level has a higher max value than 499 in the last minute. Grace period: 0 minutes. Including last 5 messages in alert notification. Configured to repeat notifications.

These are my notification settings:

add_attachment: true
backlog_items: 1
channel: #logging
color: #AA0000
custom_fields: <empty>
graylog2_url: https://url.../
icon_emoji: <empty>
icon_url: https://icon...
link_names: true
notify_channel: false
proxy_address: <empty>
short_mode: false
user_name: Graylog STAGING
webhook_url: https://hooks.slack.com/services/xxx/yyy

This is the log message with level: 500 I want to see in Slack: image

This is what I see in Slack: image Not the log message which caused the alert, but one before or after it.

Is it a config issue or a bug?

derFunk commented 7 years ago

I was able to work around what I described above by setting alerts on a new stream which dedicatedly only log errors.

Then I'm getting the current log message which caused the alert (obviously because there's not happening a lot inside stream (?) ).

I'm not sure if this is by design or a bug, but it seems like when I use a stream with a lot of messages, the backlog message filter is not correctly set to the time where the log message occured which triggered the alarm.

Is my setup described above not what this plugin should be able to handle? (Have lots of log message, then set a condition to a field and when it's met: alert).

That the causing log message is not sent, can it be a timezone issue? (we're CEST/UTC+2). Maybe it's a wrong filter when calling getMatchingMessage()? (I just had a quick look).

Aenima4six2 commented 6 years ago

@derFunk Can you confirm this issue is happening in the latest release please?

derFunk commented 6 years ago

Yep, but I'll need some time to put my hands back on it, I'll report back as soon as I have a result. Thanks!

Aenima4six2 commented 6 years ago

Awesome, thanks @derFunk . Please reopen the issue if you are able to reproduce in the latest version.