[Security Solution] Consolidating Rule Management with Alerting Framework

[banderror]

Summary

Elastic is going to build a Kibana application for unified rules and alerts management, and @elastic/response-ops-ram team will be working on that in the future. This app will be showing Security, Observability, and Stack rules in a common table and allow to render some solution/domain-specific UI in it (for example, by clicking on a rule it could show a Rule Details flyout that could be a domain- or rule type-specific piece of UI).

On the other hand, in Security we want our Rules table to be consistent with the unified rules table in terms of UI/UX patterns and components used so that the user experience is not too much different for users when they work with rules in Security and this unified app. However, we still need to be able to present our domain-specific UIs and include our domain-specific logic into the Rules table in Security.

So the plan is to:

• Start reusing components from the Alerting Framework and build our table on top of them. It won't be a "framework"-like approach where we'd be importing some generic rules table with logic and then customizing it by specifying props, injecting dependencies, etc. Instead, it would be a "library"-like approach where we'd be using lego blocks provided by the Framework, but we'd be combining them as we want on our side.
• Start moving towards Alerting Framework API endpoints and deprecating our own Detections API endpoints if/where possible.
  • Primarily it's about reading/searching rules via the Alerting API - let's think about how we can do that first.
  • We can probably keep our own CRUD and actions endpoints - that's debatable.

How can we use Alerting API for reading rules

Right now it's not possible to just start reading rules via the Alerting API endpoints. The reason is that we have our domain-specific code in them:

• Sidecar saved objects that we query, process, and include the processed info into the rules being returned
  • siem-detection-engine-rule-execution-info sidecar SO that we merge with some info from the rule SO and return in the rule as execution_summary property.
  • Legacy rule actions sidecar SOs (two SO types I think).
• Transformation of the Rule data model that's returned by RulesClient into our Rule domain model that we return from our HTTP API endpoints.
  • The data model is closely tied to the persistence layer. It contains raw params object as it's stored in .kibana index. It lets the user know that there is some params object. It doesn't make sure that required params are there if they are not stored in _source, etc.
  • The domain model doesn't expose any details about persistence. It shows rule type-specific fields as rule.riskScore rather than rule.params.riskScore. It normalizes some fields where we need it (e.g. sets default values for missing params).
  • This decouples what we expose to the user from how we store it in Elasticsearch, gives us room for safe data migration and backward compatibility, etc.
• Server-side on-read validation of the rules being returned. This gives us confidence that if we returned a rule, it's correct and valid and won't break our app on the client-side.
• Most likely a lot of other domain-specific logic.

So to proceed, we'll need to:

• get rid of our domain-specific code (e.g. sidecar SOs)
• and/or move our domain-specific code to the Framework (e.g. transformation to the domain model)

Sub-tasks

### Experiment with adding reusable components from Alerting to the Rules table in Security
- [x] Build a POC for adding UI for snoozing rules to the Rules table ([PR](https://github.com/elastic/kibana/pull/137430)) 

### Get rid of our sidecar saved objects and remove all the related code (SO types, logic, etc)
- [x] Rule execution info sidecar SO https://github.com/elastic/kibana/issues/130966
- [ ] Legacy rule action sidecar SOs https://github.com/elastic/kibana/issues/146288

### Migrate to Alerting CRUD endpoints
- [ ] Research and find out what other domain-specific logic we have in our Security endpoints.
- [ ] Add support for on-read transform (normalization) function to rule types in the Alerting Framework.
- [ ] Add support for on-read validation function to rule types in the Alerting Framework.
- [ ] Decouple rule creation/editing UI from actions creation/editing UI. Create or edit actions from a separate page. Reuse the existing Alerting Framework's UI and API for creating and editing actions. Create a new set of dedicated endpoints in the Alerting API for managing actions, if needed. Stop exposing rule actions from the Detections API.
- [ ] Stop exposing exception lists from the rule via the non-exceptions related Detections API endpoints, such as rules CRUD endpoints.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)