[Security Solution] Error when filtering alerts from the Execution results tab: "Cannot find field 'kibana.alert.rule.execution.uuid' in alerts index"

[banderror]

Kibana version: main branch corresponding to 8.7.0

Summary

On the Rule Details page, Execution results tab, when you click on the filter alerts button, it shows an error:

The kibana.alert.rule.execution.uuid field is, however, mapped in the alerts index:

GET .alerts-security.alerts-default/_mapping

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[spong]

I did a quick debugging of this (this feature was just working for me on main yesterday), and this is what I'm seeing:

For this action we're using the indexPattern from the SourcererDataView to fetch the uuid field so we can add it to the global query bar, and in debugging on a fresh local deployment I'm seeing the indexPattern being returned with only two fields (_id and one other).

I continued debugging a bit and then it just ended up working and indexPattern was then being returned with all expected fields from the alerts index. I believe what ended up fixing it was a full-page refresh...

Previously, there was a warning toast that the Sourcerer would display saying that the Security Solution Data View wasn't initialized and that you needed to refresh the page, but I'm not seeing that toast anymore. Let me reach out to the threat-hunting folks and see what might've changed here as I know they've been working to adopt the core DataView API to some degree (https://github.com/elastic/kibana/pull/139671).

edit: Appears to be related to some of these recent changes. They're gonna dig in a little further and maybe bring that toast back. Will update this issue once we hear back 🙂

[banderror]

@spong What makes us depend on that SourcererDataView? Can we remove this dependency and just add a simple filter kibana.alert.rule.execution.uuid: value?

[spong]

So we use the SourcererDataView to fetch the indexPattern (DataViewBase) and field (DataViewFieldBase) that way we can use the core kbn-es-query buildFilter() function to build the filter that then gets set using the global filterManager.

https://github.com/elastic/kibana/blob/21de750b3cff4d4ad27d07c77135327e644b3bee/x-pack/plugins/security_solution/public/detection_engine/rule_details_ui/pages/rule_details/execution_log_table/execution_log_table.tsx#L271-L289

We could just construct the Filter manually (not using the core functions) as all it looks like is:

{
  meta: { alias: null, negate: false, disabled: false },
  query: {
    match_phrase: { 'kibana.alert.rule.execution.uuid': 'e71ce319-58fe-460a-8de7-31216254128a' },
  },
}

however I've ran into issues in the past doing this where internal core changes would end up breaking the filter. Perhaps those were one-off's, and so long as we use the appropriate Filter interface they should update our code when they make changes, but generally speaking anytime we're working with the global query bar/filters/timepicker, I've found it's best to just use their supplied helpers.

The SourcererDataView not being initialized is still an issue though and will need to be fixed, so I would opt to rely on that upstream fix to resolve this, but if you'd like I can take a peek into just manually constructing the Filter and see if that'll suffice.

[banderror]

@spong Got it, thank you!

The SourcererDataView not being initialized is still an issue though and will need to be fixed, so I would opt to rely on that upstream fix to resolve this

I agree 👍

if you'd like I can take a peek into just manually constructing the Filter and see if that'll suffice

I'd postpone this at least until after the FF -- when we could groom the bugs backlog and pick something from it.