email alert content parameters show '< VALUE > MISSING'?

[Canon88]

Hi, guy: I met a problem, when I used the email alert, adjust the alarm template. But the email content parameters show '< VALUE > MISSING' is this why? Example: my configuration:

name: elk-flatline-test
type: flatline
es_host: x.x.x.x
es_port: 9200
index: logstash-test_log-*
threshold: 3
timeframe: 
    minutes: 1
filter:
- query:
    query_string:
        query: "status:ERROR"
use_count_query: true
doc_type: logstash-test_log
alert:
- "email"
email:
- "my@email.com"
alert_subject: "Alert {0} at {1}"
alert_subject_args:
- status
- date
include: ['alert_text']
alert_text: 
    this is {0} at {1}
alert_text_args:
- date
- status

Email alarm content:

title:
Alert <MISSING VALUE> at <MISSING VALUE>

body:
this is <MISSING VALUE> at <MISSING VALUE>

An abnormally low number of events occurred around 2016-09-03 12:49 CST.
Between 2016-09-03 12:48 CST and 2016-09-03 12:49 CST, there were less than 3 events.

@timestamp: 2016-09-03T04:49:16.077075Z
count: 0
key: all

[Qmando]

The only fields you can use in alert_subject_args and alert_text_args are count and @timestamp.

Are you trying to make an alert when there ARE error messages? If you say what you are trying to achieve here I can help.

[Canon88]

Yeah, I want to try when flatline alarm, Send mail to custom alert_text_args, alert_subject_args parameter error message.

[Qmando]

You know that flatline means when there are LESS than 3 events? That's what you want to do? The filter you have is only matching ERROR documents, not filtering them out. Perhaps you want the inverse of that, alert when less than 3 NON-error messages?

For flatline alerts, you can't always access fields from the data because it can be trigger on the lack of data. With flatline OR frequency, when using use_count_query, you can't access any fields.

[Canon88]

I want to express meaning. I want to know. How can let alert_text_args, alert_subject_args work. At the moment, I received an email in < VALUE > MISSING. I don't know what is going on. I hope you can help me, thank you. I'm sorry, my English is not very good.

[Canon88]

type: flatline
threshold: 3
timeframe: 
    minutes: 1
filter:
- query:
    query_string:
        query: "status:ERROR"

For this configuration, when 1 minutes, status: ERROR less than 3. The alarm will occur. This is my understanding.

[Canon88]

@Qmando Thank you for your answer. I think I've found the problem, because use_count_query = true, I disable use_count_query normal. Thank you again!

[bean5]

I ran into this as well. So the solution is to set use_count_query: false explicitly or to not set it at all?

[bean5]

Hmm, I specified 'use_count_query: false' and I still get <MISSING VALUE> coming through.

[Qmando]

If you use the default alert text with --debug, the match dictionary will get dumped, so you will see exactly which fields you can use. It prints them out line by line as "Key: Value".

If you are using flatline alert, the alert can be generated on ZERO documents, which means no fields would be available.

[bean5]

So query_key grouping is done by ES? For some reason I though ElastAlert was caching the set of query_key values and using those to detect when flatline threshold is met _when querykey is set. I was using the query_key as a value in alert_text_args.

[Qmando]

Ah! Elastalert does the grouping. This one is not very intuitive but that value can be accessed with "key" and not whatever query_key is set to.

https://github.com/Yelp/elastalert/blob/master/elastalert/ruletypes.py#L476

Several people have run into this exact issue several times recently, so I'll add some code to use the actual field name.

[bean5]

That would be awesome.

You might want to consider being backwards compatible if you think many are using "key" instead of the actual key's name.

This one change would really make flatline more in-line with how other rules work. Not being able to access that key's name makes this particular alert less intuitive when query_key attribute is set in the rule.

Looking forward to that code.

[cfbao]

Just to confirm, when using flatline rule with query_key, I should put the literal word "key" in alert_text_args? like this:

query_key: Instance
alert_text: |
  Instance {0} flatlined
alert_text_args:
  - key

and it's still not possible to use the actual query key name ("Instance" in the example above)?