[Security Solution] Warnings in rule filters on the Rule Details page: "Field does not exist in current view"

[banderror]

Related to: https://github.com/elastic/kibana/pull/177081 Kibana version: 8.14.0-SNAPSHOT

Summary

If you create a rule with a filter, such as host.name: "some-value" AND host.os.family: "windows", then:

• On the Rule Creation page, during rule creation, the filter will be displayed correctly.
• On the Rule Details page, after saving this rule, it will be displayed with warnings, e.g. host.name: Warning AND host.os.family: Warning.
• On the Rule Editing page, if you edit this rule, it will be displayed correctly again.

Rule Creation:

Rule Details:

Steps to reproduce

0. Launch a clean Kibana + ES environment.
1. Create some valid indices with source events. Locally, one easy way to do this would be using the resolver_generator script that generates fake endpoint events (events generated by Endpoint Security aka Elastic Defend): node x-pack/plugins/security_solution/scripts/endpoint/resolver_generator.js --node http://elastic:changeme@127.0.0.1:9200 --kibana http://elastic:changeme@0.0.0.0:5601/kbn --numHosts=5 --numDocs=2.
2. Create a new custom rule. Keep the default set of index patterns if you used the resolver_generator script. Otherwise, point the rule to the indices you created on the previous step.
3. Enter * as the rule's query.
4. Add a rule filter, for example host.name: Host-avy6d0956e AND host.os.family: windows (use any values from your source data).
5. Notice that the filter is displayed without any warnings, and the field values in the filter are clearly visible.
6. Save the rule.
7. On the Rule Details page, notice that instead of the field values Warnings are displayed.

Expected behavior: on the Rule Details page there shouldn't be any warnings in rule filters, when we know that source events with the field values used in the filters exist. Field values should be displayed instead of warnings, just like on the Rule Creation and Editing pages.

Hypothesis

Maybe the bug is caused by the fact that on the Rule Details page we use a data view that includes only the .alerts-security.alerts-<spaceid> index:

The filter's UI component tries to find the filter's fields and their values in this data view, and doesn't find them because there are no alerts created with these fields yet. You can check in Discover that indeed, there are source events with those fields, but there are no alerts:

Source events:

Alerts:

So the fix would be to use on the Rule Details page a data view that would correspond to the list of index patterns or the data view of the rule, instead of the data view pointing to the alerts index of the current Kibana space.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[shayfeld]

Hi @banderror :)

How is this issue progressing? It is not possible for my SOC team to see the value inside the filters in definition screen.

[banderror]

Hi @shayfeld, thanks, I'm raising the priority for this one. Although it's not a commitment, there is a chance that we will have some freed up resources to work on that closer to the end of this year.

[nikitaindik]

Hey @shayfeld and @banderror! I've investigated the bug and figured out what causes the issue. The issue affects only the filters that have "AND" or "OR" conditions.

In our app both index patterns and data views are represented as "data view" objects. TS type is either DataViewBase or DataView.

On the Rule Details page we are using an incomplete DataViewBase object that doesn't have a value for id field.

When the filters UI component renders it checks if filter is applicable to a data view (index patterns). It does this by verifying that "data view" id is equal to filter's meta.index value. Normally both data view id and filter's meta.index have a value that is a stringified index pattern, like "logs-*,events-*". But on the Rule Details page we are using a "data view" that doesn't have an id, so the check fails and a warning is shown.

Why it works on Rule Editing page, but not on Rule Details page?

Rule Editing page creates complete DataView objects with id field present.

Rule Details page creates incomplete DataViewBase objects without id.

Possible fix

I noticed this issue while working on one of my previous tasks and made a branch with a fix. In short, the fix is creating DataView objects instead of DataViewBase. The fix has yet to be properly reviewed, tested and merged. The team is currently busy with the Rule Immutability/Customization epic, so I can't really give a good estimate for the release version.

[banderror]

Thank you @nikitaindik for documenting your findings here, this will be useful when we get back to finalizing the fix 👍