[Security Solution] Notification actions: duplicated nested + flattened properties in `parameters.filters` in action payloads

[banderror]

Related to: https://github.com/elastic/kibana/issues/163844 Kibana version: broken in 8.9.0 through main; works in 7.x

Summary

A rule with a simple filter like this one:

  "filters": [
      {
          "meta": {...},
          "query": {
              "match_phrase": {
                  "host.os.family": "windows"
              }
          },
          "$state": {...}
      }
  ],

And a webhook action with the following payload:

{{#context.alerts}}
{{.}}
{{/context.alerts}}

Generates proper alert documents with correct kibana.alert.rule.parameters.filters values in them:

            "filters": [
              {
                "meta": {...},
                "query": {
                  "match_phrase": {
                    "host.os.family": "windows"
                  }
                },
                "$state": {...}
              }
            ]

But sends incorrect filters in the webhook's payload (notice how the flattened field name gets unflattened and injected as a nested object to match_phrase):

          "filters":[
            {
              "meta":{...},
              "query":{
                "match_phrase":{
                  "host.os.family":"windows",
                  "host":{"os":{"family":"windows"}}
                }
              },
              "$state":{...}
            }
          ]

Some users rely on correct filters[0].query values in their systems and automation, and they can't use these broken values anymore because they are not correct Elasticsearch queries.

Details

Could this be related to https://github.com/elastic/kibana/issues/163844? I think the bug is likely on the Detection Engine side in the formatAlertForNotificationActions function that is called before we pass the alert data to the Alerting Framework for scheduling actions:

https://github.com/elastic/kibana/blob/b93868ec3886b2acb3af18735a1602730dcd3811/x-pack/plugins/security_solution/server/lib/detection_engine/rule_actions_legacy/logic/notifications/schedule_notification_actions.ts#L53-L62

This function implements expected behavior, but it could be that it incorrectly handles this edge case with filters.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[Dariaus999]

@yctercero - We would like your help in addressing a customer concern. Despite our best efforts, we have been unable to locate any mention of a fix in the 8.14 & 15 release notes or elsewhere. Additionally, the issue itself appears to remain open. Is it possible to confirm whether this patch was already included in versions 8.14 and 8.15?

[yctercero]

@Dariaus999 - I apologize for not updating the labels on the ticket. Once a ticket is addressed, a fixed label is added to it after which QA verifies the fix.

I've added it to our 8.16 bug list and deprioritized a few others that were on there to do our best to get to it.