elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.6k stars 8.22k forks source link

[Security Solution][Detection Engine] Optimize large value list evaluation #192766

Open marshallmain opened 1 month ago

marshallmain commented 1 month ago

Parent issue: https://github.com/elastic/security-team/issues/10106

Evaluation of large value list exceptions (see https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/utils/search_after_bulk_create_factory.ts#L160-L165) in the current implementation requires full source documents to be loaded from Elasticsearch before we can determine if the document should be excluded by a value list exception. When source documents are large, this can put unnecessary memory and CPU pressure on Kibana and Elasticsearch. We should instead look to initially load only the field(s) from each document necessary to evaluate the value list exceptions and load the full source document once we know for sure that an alert should be generated.

elasticmachine commented 1 month ago

Pinging @elastic/security-detection-engine (Team:Detection Engine)