search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.41k stars 1.07k forks source link

Alerts on some messages are missing #4415

Closed jasonkeller closed 6 years ago

jasonkeller commented 6 years ago

Expected Behavior

I have an alert configured on Field Content Condition, to fire repeatedly with no grace time when specific messages are encountered. This is to set an automated change log process in motion. I would expect then that for each message that is encountered in the stream meeting this criteria, gets an alert sent out.

Current Behavior

Currently, only the first message from within the 1 minute interval is actually alerted upon. The others are simply ignored. For example, I have five firewalls updating at two different times. I'd expect five alerts, but only receive two alert emails (the first one from each 1 minute time window).

Steps to Reproduce (for bugs)

Configure a stream to alert based on a field content condition with no grace time, set to repeat notifications. Send multiple messages meeting this criteria into the stream a few seconds apart within the same minute boundary.

Context

Consequently, our change log infrastructure is not registering all the updates that it should be, as there is not an alert for each firewall update.

Your Environment

jalogisch commented 6 years ago

@jasonkeller to be honest that works like it was designed. If you follow this in the documentation you can read:

Graylog alerts are periodical searches that can trigger some notifications when a defined condition is satisfied

It will not triggered on any found condition, it will be triggered because the condition is true in the search period (default one minute after the last search was finished).

You like to add the feature to trigger on every occurrence of the condition. Did I get it right?

jasonkeller commented 6 years ago

If this is the expected current behavior, then yes this would be a feature request instead of a bug report, to trigger an alert on every occurrence of a condition. Thank you @jalogisch

lennartkoopmann commented 6 years ago

The alerting engine is not built for this kind of high-cardinality alerting. We have a feature in development for 3.0 that might help with this if you don't need the alerts triggered in near-realtime.

Did you also consider using the outputs functionality? You could catch all messages that you want to alert or trigger an action on in a stream and forward them to a script or service in near-realtime for further processing.

jasonkeller commented 6 years ago

@lennartkoopmann for that particular use case we would not need the alerts triggered in real time, as the change management process will simply use the timestamps embedded in the messages themselves, so temporally speaking some delay prior to getting those alerts will not be an issue.

I've considered using an output, but that would make a huge change for some of our developers as the only hook they have right now into the other system has typically been via email. But I'll ask anyway :).