Creating an alert with multiple indices and selecting a time field

[mdefazio]

Not sure if this is a bug or not, so apologies for any confusion. More of a question I suppose.

If I have 2 indices I've selected, but the time field selected is only associated with one of those indices, how (or why) does it validate?

In this example, customer_birth_date is only associated with the ecommerce index. And if I were to remove the ecommerce index from the list, the time field then also gets reset since that value is no longer available (which makes sense).

My initial thought was that the time field would list the common fields between the indices and not an aggregation of all across the two.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

I struggle with these field pickers - mainly with the index action. For testing, I often want to create a new index via the action, which works, but then I can't provide a time field, since the index doesn't exist and so there are no fields to pick. So, ideally, I'd like a editable combo box listing "existing fields", but also allow me to enter a new one myself.

In that vein, I'm not sure listing fields as a union across all indices is wrong. But could certainly be confusing. In this case with the index threshold, documents in an index that doesn't have the time field would be ignored. Which definitely seems wrong, or at least worth warning the user about.

Also, I'd guess my issues with dealing with non-existent indices isn't a problem a customer is going to see. So more weight on providing the intersection of the fields vs union.

Presumably this also applies to the field picker for the aggregation fn's (avg, max, etc) and the "top-n" aggregation field. The top-n one may be interesting, if the agg ended up returning a bucket for "null" entries for the field - not sure if it does or not. But the aggregation fn's are like the timestamp - if an index didn't have that field, I think all the docs in the index would be ignored.

We'll have to look into the API we're using to get this info - perhaps there's an option to get the intersection vs the union, if that's what's happening ...

[pmuellr]

We had a discussion on this issue while doing some issue triage, and we're not really sure what the expected behaviour should be.

This feels like something that probably comes up in other places within Kibana, where the user could enter an index pattern (ES-style or Kibana Index Pattern saved object) that ends up referencing multiple indices. This could happen even with an "index name" that looked like a single index, but was really an alias over multiple indices.

So I guess the question is, is there some kind of "common Kibana behavior" here that we need to align with?

Also noted in the call, one of the problems this could cause is alerts being missed if the customer used multiple indices, one didn't have the time field (or other fields referenced in the alert - the threshold field, the group-by field, etc), presumably that index would be ignored, and "alerts could be missed". Since it's not clear that at creation time we can do a whole lot, maybe we need to provide more diagnostics for our runs. Eg, the queries we do for index threshold do end up returning the index the documents were found in, so we could somehow make these available for later diagnosis.

[pmuellr]

Going to close this, as we don't really have anything thoughts on making this directly actionable at the moment.

Sort of feels to me like the note at the bottom of the previous comment ^^^ on adding additional diagnostic data (the indices where the documents were found when the alert ran it's query) is likely something that could add fairly easily (as a new event log document, probably), if we end up needing to.