search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.66k stars 8.23k forks source link

[Security Solution][Detection Engine] duplicated alerts for new terms rule #174148

Open vitaliidm opened 10 months ago

vitaliidm commented 10 months ago

Describe the bug: When new terms rule configured with large look back times (x2-4 times greater than rule interval) and multiple documents present in source index that triggers alert, duplicate alerts can possibly appear.

It happens due to at the third phase of new terms rule execution where the oldest document that has new terms combination will be searched.

Phase 3: Any new terms from phase 2 are processed and the first document to contain that term is retrieved. The document becomes the basis of the generated alert.

For example, if look back interval 30m and scheduled interval - 10m , it means rule evaluates newly appeared terms 4 times. Which means, rule will perform 4 searches for the oldest document, to make it an alert. If there are multiple documents with the exact same timestamp or very close one, each of these 4 evaluations might pick up a different "oldest" document. What will results in multiple duplicated alerts. Since deduplication logic for new terms rules includes: event id and new terms values.

Having multiple shard in source index might contribute to having non-deterministic results in search(issues with cluster, node availability, heavy load)

Kibana/Elasticsearch Stack version: 8.11+(but affects every version with new terms rule)

Any additional context (logs, chat logs, magical formulas, etc.):

Proposed solution is to avoid non-deterministic alert creation for new terms rule. We can't guarantee, each time the same document will be returned as oldest one in the third phase. So, that solution can be to not include document's id into alert id. Instead, we may just limit id as hash of new terms + interval on which it was found for that specific rule.

Related issues

SDH internal

elasticmachine commented 10 months ago

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine commented 10 months ago

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine commented 10 months ago

Pinging @elastic/security-detection-engine (Team:Detection Engine)