[Security Solution][Detection Engine] ML Rule forms have incorrect autocomplete fields

[rylnd]

Summary

There are several fields that leverage autocomplete on the rule forms:

• Severity override
• Risk Score override
• Timestamp override
• Rule name override
• Custom highlighted fields
• Alert Suppression "suppress by" fields

However, in the case of ML rules, there is no underlying "data source" associated with the rule (at least as far as the UI is concerned), which leads to some interesting behaviors:

1. When creating a new ML rule, any autocomplete components are populated with fields from the default security index pattern, since that's loaded/populated before the ML rule type is selected. At best, these fields are stale relative to the actual ML rule's data source(s), but at worst (and in the typical case) hese fields are absent from said data source.
2. When editing an ML rule, any autocomplete components no longer have "possible" values, and only contain the fields that were selected during rule creation.

In both of the above scenarios, workarounds are limited since it isn't possible to manually add a field to those autocomplete components (as far as I am aware).

Steps to reproduce:

1. Create an ML rule
2. Edit the rule
3. Attempt to add a field for Rule Name override

Expected behavior: At the very least, I should be able to manually add fields to these components. Ideally, that process is aided by autocompleting from the ML indices directly, or else a static list of ECS fields.

Screenshots:

Any additional context: This behavior looks to be as old as ML rules, which predate many of the autocomplete fields the rule creation/edit UI use.

This also relates to the recent issue with ML Rule Preview, which (with hindsight) I would now classify as a symptom of this broader issue: preview was based on the presence of an index argument (aka a data source), for which ML rules don't have an official concept.

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[rylnd]

Update: after some discussion we've elected to spike out the following: ML rules will use the alerts index fields for its autopopulation. If that's relatively straightforward, it will satisfy the second issue listed above, and we can split off a separate "ML autocomplete doesn't include anomaly fields" issue as needed.

[rylnd]

After some discussion I was pointed to the fact that the useRuleIndices hook already contains this logic (most of which was added in https://github.com/elastic/kibana/pull/133494), and is used in the expected places on the rule UI.

There are currently two issues with this implementation:

1. If the corresponding ML job is not installed, the hook currently falls back to using the default security index pattern. In the case of ML rules, this is the incorrect behavior as those indices have little/no relation to the anomaly indexes.
2. If the ML Job is not installed, we have no index to retrieve fields from, and no obvious fallback mechanism. This is potentially where @jgowdyelastic's "anomaly fields API" could be used.

[rylnd]

Update: we have addressed the majority of concerns in rylnd/kibana#9, and that will soon be merged to main via https://github.com/elastic/kibana/pull/181926.

@yctercero I think once that's merged we can close this, and perhaps open a broader ticket for a more formal integration with ML?

[yctercero]

Amazing! That sounds good. @approksiu and @ARWNightingale are working through new create rule flows. It may be great to sync up on the particularities of ML.

[rylnd]

This was addressed (in part) by #181926, which added a new hook useRuleFields to retrieve the correct fields for a particular rule (including ML rules). It has only been applied to the ML Suppression fields, however, and so that new hook still needs to be leveraged by the other relevant UI fields.