Spike alerting email only shows one log even if thousands hit the rule

[yxz900]

My spike rule configuration is as follows:

name: conn lost spike type: spike index: logstash-* threshold_cur: 50 timeframe: . minutes: 5 spike_height: 3 spike_type: "up" filter: - query: query_string: query: 'tags: "xxxxx"' include: ["hostname", "syslog_timestamp", "message", "mdt"] alert: - "email" email: - "myemail" from_addr: "elastalert@mymachine" alert_subject: "conn lost spike" alert_text_type: alert_text_only alert_text: "conn lost from {} to {} at {}" alert_text_args: - hostname - mdt - syslog_timestamp

Elastalert run log had the following: INFO:elastalert:Queried rule conn lost spike from 2020-09-17 20:33 AEST to 2020-09-17 20:48 AEST: 4618 / 4618 hits INFO:elastalert:Sent email to ['myemail'] INFO:elastalert:Ran conn lost spike from 2020-09-17 20:33 AEST to 2020-09-17 20:48 AEST: 4618 query hits (5 already seen), 1 matches, 1 alerts sent

I expected to see 4618 messages in the email. However I only received one: conn lost from xxx to yyy at 2020-09-17T20:00:06+10:00

Can someone please let me know what I'm doing wrong? Many thanks!

[vtdat]

You might want https://elastalert.readthedocs.io/en/latest/ruletypes.html#aggregation and https://elastalert.readthedocs.io/en/latest/ruletypes.html#summary-table-fields

Be aware of email size https://elastalert.readthedocs.io/en/latest/elastalert.html?highlight=max_aggregation#configuration.

[yxz900]

Thanks for your swift reply. I think summary_table_fields is only needed if alert_text_type is aggregation_summary_only. I'll try adding aggregation. I didn't add it because it's not in the elastalert example spike files.

You might want https://elastalert.readthedocs.io/en/latest/ruletypes.html#aggregation and https://elastalert.readthedocs.io/en/latest/ruletypes.html#summary-table-fields

Be aware of email size https://elastalert.readthedocs.io/en/latest/elastalert.html?highlight=max_aggregation#configuration.

[vtdat]

Thanks for your swift reply. I think summary_table_fields is only needed if alert_text_type is aggregation_summary_only. I'll try adding aggregation. I didn't add it because it's not in the elastalert example spike files.

no, it's not required, but recommended :D

[yxz900]

Thanks for your swift reply. I think summary_table_fields is only needed if alert_text_type is aggregation_summary_only. I'll try adding aggregation. I didn't add it because it's not in the elastalert example spike files.

no, it's not required, but recommended :D

OK I'll add it to see how it goes. Thanks:-)

[yxz900]

So the default maximum number of alerts to aggregate is 10,000 and I got 4618 which is less than 10,000. Don't understand why I didn't get the alerts...