search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.51k stars 8.06k forks source link

[Security Solution] OOMs during prebuilt rules package installation #187969

Open xcrzx opened 2 weeks ago

xcrzx commented 2 weeks ago

Epic: https://github.com/elastic/kibana/issues/174168

Summary

We've seen a number of incidents with OOMs when installing the security_detection_engine Fleet package that contains a significant number of prebuilt rules (> 5000 in Serverless or > 15000 locally).

We need to find and mitigate all memory bottlenecks during package installation to unblock the release of the prebuilt rule customization epic.

### Known memory issues
- [ ] https://github.com/elastic/kibana/issues/187646
- [ ] https://github.com/elastic/kibana/issues/187975
- [ ] https://github.com/elastic/kibana/issues/188208
- [ ] https://github.com/elastic/kibana/issues/189043
elasticmachine commented 2 weeks ago

Pinging @elastic/fleet (Team:Fleet)

elasticmachine commented 2 weeks ago

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

elasticmachine commented 2 weeks ago

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine commented 2 weeks ago

Pinging @elastic/security-solution (Team: SecuritySolution)

kpollich commented 4 days ago

@xcrzx - I wonder if we could introduce another option aside from force (which causes Fleet to uninstall all previous assets before installing new ones) that would allow Fleet to skip the initial deletion of old assets and just run the bulk import operation as an optimization here.

xcrzx commented 3 days ago

@xcrzx - I wonder if we could introduce another option aside from force (which causes Fleet to uninstall all previous assets before installing new ones) that would allow Fleet to skip the initial deletion of old assets and just run the bulk import operation as an optimization here.

Yes, incremental installation might help for our use case. I'm not sure how SO import handles conflicts when writing new assets, but if we can instruct it to ignore existing saved objects or filter them before passing them to the import function, that should work.