search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.58k stars 8.21k forks source link

[Security Solution][Detection Engine] ES|QL rule type generates less alerts than expected with MV_EXPAND #197130

Open marshallmain opened 1 week ago

marshallmain commented 1 week ago

The MV_EXPAND command is in preview and was added to ES|QL for 8.11 (https://www.elastic.co/guide/en/elasticsearch/reference/8.15/esql-commands.html#esql-mv_expand). This command takes a multi-valued column and produces a separate row in the results for each value. As a consequence, the results can have multiple rows with the same _id and _index values but the DE rule logic will only create an alert for one of these rows due to deduplication. We should investigate what the correct handling is in this scenario.

elasticmachine commented 1 week ago

Pinging @elastic/security-detection-engine (Team:Detection Engine)

vitaliidm commented 1 week ago

Possible way to do this is to extend what properties we include in alert id.

Currently, for non-agg queries we include id, index, version, ruleId, spaceId

By using utilities from @kbn/esql-ast package, I believe we would be able extract mv_expand field and retrieve its value from processed result row. Then we can include it key-value object into id generator. In this case, this would allow to generate unique alert id per each "expanded" value. One corner case, we should keep in mind, user might drop or rename "expanded" field. In this case we might resort to remove all identical values from result rows and use the rest values in id generation.