[Security Solution] Create asynchronous rules import API

[banderror]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), https://github.com/elastic/kibana/issues/174168

Summary

With https://github.com/elastic/kibana/issues/180168, we're going to introduce additional logic to the import endpoint for calculating the rule source object. Some of this logic will be run once for a given import call, some of it will be run multiple times for each rule being imported. Some of it can be IO-heavy (installing the package, fetching historical rule versions and ids), some of it can be CPU-heavy (calculating a diff for each rule).

Based on our prior observations, the rules import endpoint times out when importing a large number of rules. I think the number can be around 2-3k rules. Now, with the additional logic, the endpoint is going to be even heavier and can start timing out with a lower number of rules in the ndjson payload.

In order to address this issue, our idea was to add an asynchronous rules import API. This API:

• Could be implemented via a request parameter for the existing import endpoint, or via a separate endpoint (or set of endpoints)
• Async import endpoint would respond immediately and run the import process in the background.
• Optionally, it should be possible to get the status of a given import operation.
• Optionally, it should be possible to cancel a given import operation.

We also need to figure out how switching to this async API can affect the import UI on the Rule Management page.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[rylnd]

@banderror thanks for writing this up!

I think there are a lot of outstanding questions here, but a few high-level ones that come to mind:

1. [product] Is the "import status" API a new HTTP endpoint? Is there a UI component as well?
2. [product] What are the contents of an import's status? E.g. number of rules imported, number remaining, number failed?
3. [product] Are we okay with cancellation of an import resulting in a partial import? We certainly don't have a way to do the import in a single transaction, and trying to delete any already-imported rules likely wouldn't be feasible for a number of reasons.
4. [technical] How/where to we intend to store the uploaded NDJSON file for processing? I think we can potentially use the file plugin (which cases uses for file upload), but that needs to be verified.

[banderror]

The above questions have been discussed with @rylnd and @jpdjere.