[Security Solution] Rule run is failing for a CCS EQL with event absence due to the kibana.alert.ancestors.index value multiplication in the alerts

[pborgonovi]

Describe the bug: Rule run is failing for a CCS EQL with event absence due to the kibana.alert.ancestors.index value multiplication in the alerts

Kibana/Elasticsearch Stack version: 8.15

Server OS version:

Browser and Browser OS versions:

Elastic Endpoint version:

Original install method (e.g. download page, yum, from source, etc.):

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Pre-requisites:

1. Have 2 clusters connected
2. Have multiple documents indexed on remote cluster within the maxspan of 60 seconds e.g: 5000 documents with initial timestamp and 5000 documents with timestamp +30s

   {
   "@timestamp": "2024-08-08T23:15:30.000Z",
   "host": {
     "name": "paulas-macbook-pro.local"
   },
   "agent": {
     "name": "Paulas-MacBook-Pro.local"
   }
   }

Steps to reproduce:

1. In the primary cluster, create an EQL rule with the index pattern from the remote cluster and containing event absence in the query

e.g:

index pattern: paula_eql_2:my_index_eql_1

EQL query:

sequence by host.name with maxspan=60s
[ any where agent.name == "Paulas-MacBook-Pro.local" ]
![ any where agent.name == "host-test"]

Current behavior: When the rule runs and alerts are genetared, kibana.alert.ancestors.index value is being multiplied repeatedly and it's causing a parse error and run failure.

It can be observed that the value of _index field when the query is directly ran is multiplied repeatedly:

From .internal.alerts-security.alerts-default-* index:

Expected behavior: kibana.alert.ancestors.index value should NOT be multiplied

Screenshots (if relevant):

https://github.com/user-attachments/assets/bac59b9a-c728-4ef9-aa7b-c6c90f00c129

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context (logs, chat logs, magical formulas, etc.):

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[yctercero]

Thanks so much @pborgonovi ! Could you confirm if this is happening in 8.14 or if it's a regression in 8.15?

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[pborgonovi]

@yctercero I've validated 8.14 as well and the results were the same:

Run failure:

kibana.alert.ancestors.index value multiplied repeatedly:

https://github.com/user-attachments/assets/f7b6ef71-3fc8-4a17-86b0-d8da2772f30d

I have observed the other fields related to kibana.alert.ancestors are duplicated in the alert:

Same duplication seems to occur to kibana.alert.ancestors.index and this value is already returning multiplied repeatedly as we can observe by the query response:

I have also verified the EQL query WITHOUT event absence and it seems the issue doesn't occur:

Running the query directly:

However, we can observe the same fields being duplicated on the alert: