[Security Solution] Migrate security rules encrypted saved objects to new schema with `ruleSource` field (BLOCKED)

jpdjere commented 4 months ago

Epics: https://github.com/elastic/security-team/issues/1974 (internal), https://github.com/elastic/kibana/issues/174168 Depends on: https://github.com/elastic/kibana/issues/180141 Blocked by: https://github.com/elastic/kibana/issues/187651, https://github.com/elastic/kibana/issues/50216, https://github.com/elastic/kibana/pull/183603#issuecomment-2151513105 Needed for: https://github.com/elastic/kibana/issues/180126

Summary

Use a (currently non-existing) rule migration mechanism provided by the Alerting Framework to migrate detection rules to a new schema that contains the new ruleSource field.

In Security Solution, as part of the rule customization epic, we need to change the rule parameters from:

type RuleParams = {
  immutable: boolean;
}

to

type RuleParams = {
  rule_source: {
    type: 'internal' | 'external'; // <- this can be derived from the existing 'immutable' field
    is_customized: boolean;
  }
}

Semantically, the fields have similar meanings; both the old field and the new field will be used to distinguish prebuilt detection rules from custom rules created by users. However, the new field allows for more flexibility and enables us to build rule customization features on top of it.

Proposed solution

Initially, we proposed to use the Model Version API for this migration in a POC, but the proposal wasn't accepted by the ResponseOps team.

At the moment, we don't have an idea what this solution should be. We depend on the ResponseOps team here, the problem is being tracked in https://github.com/elastic/kibana/issues/187651 by us and in https://github.com/elastic/kibana/issues/50216 by the ResponseOps team. We can contribute to the design of this mechanism, propose any solutions, or open an RFC.

Useful links

ESO model versioning example: https://github.com/elastic/kibana/tree/main/examples/eso_model_version_example
SO migrations on serverless docs: https://docs.elastic.dev/kibana-dev-docs/tutorials/saved-objects#saved-object-migrations-on-serverless
Implementation PRs
- https://github.com/elastic/kibana/pull/166302
- https://github.com/elastic/kibana/pull/167705

elasticmachine commented 4 months ago

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine commented 4 months ago

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine commented 4 months ago

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

banderror commented 4 months ago

Blocked by ResponseOps in https://github.com/elastic/kibana/pull/183603

elastic / kibana