[Security Solution] Stream-based installation of the package with prebuilt rules

[banderror]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), https://github.com/elastic/kibana/issues/174168 Related to: https://github.com/elastic/kibana/issues/187646

Summary

The Fleet team won't be able to implement stream-based package installation by mid October, which means we will need to implement it on our side to complete Milestone 3 in time.

Rough plan:

1. Introduce a new endpoint in Security Solution for detection rule installation or reuse the existing bootstrap endpoint. The key point is that the implementation will be entirely on the Security Solution side.
2. Copy the existing package installation logic from Fleet and strip out all code not related to saved object installation.
3. Rewrite the saved object installation process, switching from savedObject.import to savedObject.bulkCreate for better memory efficiency.
4. Implement incremental saved object installation without deleting existing objects.
5. Add Stream Support

Details

An important note here is that we'll be using the EPR API directly to fetch package information and download package content (or read from disk if it's prebundled). To ensure compatibility with Fleet, we'll reuse the package saved object type, so even if the package is installed through the Security Solution endpoint, it will still be visible in the Integrations UI. The detection rules package will remain installable and upgradeable via Fleet's UI, but this will not be the recommended method. In Security Solution, we'll exclusively use the new installation endpoint.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)