search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.42k stars 1.07k forks source link

Send alert for every message that hits stream #4132

Closed ianling closed 7 years ago

ianling commented 7 years ago

I would like to be able to send an alert for every single message that hits a stream.

Expected Behavior

If 30 messages hit a stream over the course of 1 minute, I want to receive 30 separate alerts.

Current Behavior

If 30 messages hit a stream over the course of 1 minute, I receive 1 alert.

Possible Solution

If the grace period of an alert condition is 0 minutes, send alerts as soon as individual messages come in.

Steps to Reproduce (for bugs)

  1. Create stream that matches on certain messages.
  2. Create a Message Count alert condition for the stream if there are more than 0 messages in the last minute, with a grace period of 0 minutes.
  3. Rapidly send alerts to the stream.
  4. You will only receive one alert.

Context

We monitor the status of many BGP peering sessions on many switches. If something goes horribly wrong, we might lost 30 BGP peering sessions at once, meaning that 30 syslog messages get sent to Graylog and end up in our "BGP Peering" stream.

We only get one alert, which is not very useful and contains little information, especially when 5 messages might have come from one switch, while 2 messages came from a different switch, 12 came from another, and so on.

If each message send an individual alert, then it's as easy as sorting by subject and you can quickly see that only 5 switches are involved.

Your Environment

joschi commented 7 years ago

@ianling Alerts are implemented as regularly scheduled search queries which is why there currently is no way to alert users on every single message but only on a set of messages received in the time since the last search query.

DerPhlipsi commented 7 years ago

Hey @ianling,

you could try to combine these links: http://docs.graylog.org/en/2.3/pages/plugins.html (Graylog Documentation) https://github.com/Graylog2/graylog2-server/blob/master/graylog2-server/src/main/java/org/graylog2/alerts/FormattedEmailAlertSender.java (Send E-Mail with Java like the Graylog Notification does) https://marketplace.graylog.org/addons/8eb67dc0-b855-455c-a37f-0fa8ae522854 (Example Output-Plugin)

If you are a bit known to Java it should be easy to implement a simple Output-Plugin that sends an E-Mail for each message it should output :) I sadly do not have time to write one right now, but I'll need one myself in the near future ^^

Greetings - Phil

ianling commented 7 years ago

@joschi - That makes sense, thanks for the response.

@DerPhlipsi - I'll see what I can come up with, thanks for the links!

tristanlatr commented 6 years ago

Hi,

I'm facing the same problem and wonder if you guys have came up with a solution !

Thank you !