[Security Solution] Improve rule diff algorithm performance by splitting diff calculation into async chunks

[jpdjere]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), https://github.com/elastic/kibana/issues/174168

Summary

• The calculation of rule fields diffs in the /upgrade/_review endpoint is a computationally expensive synchronous process that blocks the event loop during its calculation.
• In order to avoid blocking the event loop and improve server-side performance, we can split diff calculation into async chunks that yield control back to the event loop after a number of them are executed.

• A solution could look like so (PSEUDO):

      for (let i = 0; i < upgradeableRules.length; i += chunkSize) {
          const chunk = upgradeableRules.slice(i, i + chunkSize);
  
          chunk.forEach(({ current }) => {
              const ruleVersions = ruleVersionsMap.get(current.rule_id);
              invariant(ruleVersions != null, 'ruleVersions not found');
              const diffResult = calculateRuleDiff(ruleVersions);
              ruleDiffCalculationResults.push(diffResult);
          });
  
          // Yield control back to the event loop
          await setImmediatePromise();
      }

Background

The above solution is based on:

1. Answer from stack overflow
2. Node.js guide: Complex calculations without blocking the Event Loop

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)