[Obs Alerting] Remove dynamic look up of data source from Log threshold, Metric threshold and Inventory rules

[benakansara]

In this ticket, we concluded to go with below option in order to prepare for v9:

• Change Log threshold, Metric threshold and Inventory rules to no longer do dynamic look-up of the data source for each rule execution

The data source selection should be provided in rule form similar to what we have in Custom threshold rule. User can create new data view or use an ad-hoc data view when creating a new rule. This data view will be used as data source in rule execution. If there is an existing rule that relies on old index settings, we will create an ad-hoc data view during rule execution when it is detected for the first time. Subsequent rule executions will use this ad-hoc data view. When visiting the rule in UI, if it is detected that the rule is using old index settings, a new ad-hoc data view will be created and will be saved.

The data view selection in rule form (reference from Custom threshold rule):

[elasticmachine]

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

[jasonrhodes]

@benakansara looking at the existing Metric and Log threshold rules, the settings that they use, what are the possible options for how those values might be set, do we know that? I want to make sure we focus on the absolute bare minimum for decoupling these rules from the settings. If they are always referencing index pattern strings, we can just store that for them (with the option to edit it added to the UI). If they can possibly be data views (I think the Log Threshold rule offered data view as an option at one point so I suspect this may be the case for some customers...), then I suppose we have to allow for data views to be chosen, and so maybe the simplest thing is exactly what you describe in this issue. I just wanted to double-check before we move forward with this. Thanks!

[benakansara]

There are some differences between Log threshold and Metric threshold rules in terms of how source information is saved, but in general, we save "id" of the source in rule saved object. This id is then used to look up the indices/data view from a different saved object. As you said @jasonrhodes, it is possible to set existing data view (now deprecated, but exists) in Log settings which will be used in Log threshold rules.

I suggested to use DataView UI in this ticket because I see we use data view pattern in many UIs (Discover, SLO, Custom threshold, ES query rule, etc.), so I feel it will be a more consistent UX than introducing something different for these two rules. I also think it will be similar effort whether we work with index pattern or utilize existing data view UI.

Just to be on the same page and correct me if I am wrong, we want to decouple the rules from the settings, but also as far as I understand, main motivation for this work now is that Log settings is going away in v9, so then the question is how do we manage existing Log threshold rules for the customers e.g. if there are existing rules tied to data view or old indices setting. This is something we need to solve in the short term first.

I understand the concerns about how this can affect our rule deprecation plan i.e. users might be discouraged to migrate, but for this, we need to define a clear deprecation policy and tools to allow migration both at UI and API level. Once we notify the users in UI/API that this rule will be deprecated, I believe users will be less willing to create new rules. One thing that we need to address about migration (there might be more, but this is on top of my mind) is that Metric threshold rule allows two thresholds in rule (warning and alert) while Custom threshold does not - do we create two rules for the user in this case, how does user know about it?

[benakansara]

@jasonrhodes Is there a reason for not removing dynamic lookup in Inventory rule? That one is tied to Settings as well.

[jasonrhodes]

I feel it will be a more consistent UX than introducing something different for these two rules

I think the clearest thing I want to communicate here is that I don't think we should spend any extra time on this goal, for these rules. If we get this for free, I also don't think we need to deliberately sabotage these rules in order to increase migration to the Custom Threshold rule, either.

I also think it will be similar effort whether we work with index pattern or utilize existing data view UI.

Got it, if that's true, and it solves a number of the other potential problems we need to address here, then it makes sense to me.

Is there a reason for not removing dynamic lookup in Inventory rule? That one is tied to Settings as well.

No reason, I think it probably was just missed in the original conversation or somewhere along the way. We should include that here as well.