[Security Solution] Avoid blocking node.js event loop in Rule Management endpoints

[maximpn]

Summary

Some rule management operations may take some large amount of time if performed on a large number of rules. Current implementation relied on microtasks by using promises but it leads to blocking node.js event loop for significant amount of time. It's bad because incoming requests can't be handled while event loop is blocked and lead to failed requests. On top of that scheduled macro tasks get delayed as well.

To solve this issue long running API endpoint handlers should break down work via setImmediate(). As minimum the following Rule Management API endpoints should be updated

• bulk actions /api/detection_engine/rules/_bulk_action
• legacy bulk actions
  • /api/detection_engine/rules/_bulk_create
  • /api/detection_engine/rules/_bulk_update
  • /api/detection_engine/rules/_bulk_patch
  • /api/detection_engine/rules/_bulk_delete
• rule export /api/detection_engine/rules/_export
• rule import was addressed in https://github.com/elastic/kibana/pull/177159

Useful links

• MDN Microtask guide
• MDN Microtask guide in_depth
• What the heck is the event loop anyway? | Philip Roberts | JSConf EU

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

@maximpn I don't think this is the correct link:

rule import was addressed in https://github.com/elastic/kibana/pull/17715