Open Inkromind opened 7 years ago
sathishdsgithub
commented
7 years ago @inkromind
Please set the realert:0 and see if you stop getting duplicate emails.
also, can you set the num_events to some value and see if you get the alert email properly.
Inkromind
commented
7 years ago @sathishdsgithub With realert to 0 we get multiple alerts for similar matches, which is not something we want. We first had this and then we also got duplicate mails.
Isn't the num_events property only relevant when using the frequency-alert type? At least, that's what I get from the docs. We were using this before and set the num_events property to 1. Also no joy as it had the same effect as the any alert type we are using now.
That said, we have been tweaking the query over the past couple of days to filter out more false positives and somehow (we don't know why) the duplicate mails stopped... it seems that depending on which/how many results you get, the duplicate mails is triggered.
sathishdsgithub
commented
7 years ago @Inkromind
Can you share the rule here ?
sathishdsgithub
commented
7 years ago @Inkromind
Can you share the rule here ?
Inkromind
commented
7 years ago One of the rules started sending out multiple mails again. I did not change anything to the rule. This is the rule
name: Exception
index: logstash-*
type: any
filter:
- and:
- query:
query_string:
default_field: 'log'
query: 'container_name:"app" AND namespace_name:"test" AND *Exception* AND NOT "Invalid session ID" AND NOT Session AND NOT "token invalid" AND "java.net.UnknownHostException: local" AND NOT com.atomikos.icatch.RollbackException AND NOT "WARN ForbiddenException" AND NOT org.apache.camel.processor.validation.SchemaValidationException AND NOT com.atomikos.icatch.jta.TransactionImp.rethrowAsJtaRollbackException AND NOT javax.transaction.RollbackException AND NOT com.atomikos.icatch.RollbackException AND NOT "java.lang.IllegalArgumentException: No enum constant" AND NOT org.springframework.orm.jpa.JpaOptimisticLockingFailureException AND NOT org.eclipse.persistence.exceptions.OptimisticLockException AND NOT "One or more objects cannot be updated because it has changed" AND NOT "org.quartz.SchedulerException: Job threw an unhandled exception"'
realert:
minutes: 1
query_key: 'log'
aggregation:
minutes: 1
alert:
- "email"
alert_subject: 'ElastAlert: Exception'
email: "<obfuscated>"
tsboris
commented
3 years ago HI,
@Inkromind
Was this ever resolved? If it was, what was the solution?
Thanks :-)
We have setup several rules for alerts that should send a mail whenever it is triggered.
However, whenever a rule is triggered, it sends out 2 to 4 identical mails. The number of matches or hits reported does not match the number of mails send (sometimes it can, but other times there are eg 4 matches/hits with only 2 mails, or 4 mails with only 1 match/hit). Both aggregation and realert (on the log-field) are set to 1 min and work "properly" because multiple alerts are combined into a single mail (that is repeated 2 to 4 times) and for matches with the same log-filed, no new alert is sent within a minute. The number of alerts bundled by the aggregation is also irrelevant (eg 1 alert is sent 4 times, while 10 alerts are only sent twice).
All our rules are configured similary to this:
The different rules only differ in query and name.
We are running the latest ElastAlert pulled from the master branch (1 instance) with 3 ES instances. run_every is configured to 1 minute and buffer_time is globally configured to 15 minutes.
ElastAlert is however throwing a bunch of errors:
We also use the kibana-plugin from BitSensor, so if it's possible some of the errorlogs above are from that.