[Security Solution] Update rule searching to search by `rule_source` (BLOCKED)

[jpdjere]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), https://github.com/elastic/kibana/issues/174168 Blocked by: https://github.com/elastic/kibana/issues/184113 Needed for: https://github.com/elastic/kibana/issues/182573

Summary

After the introduction of ruleSource as a rule field that determines if a rule is prebuilt or not, we want to start using that field on all areas of the Security application where we retrieve rules filtered by wether they are Elastic Prebuilt or not.

Since the creation of the ruleSource will occur through a Models Version migration mechanism, we can have reliability that the ruleSource is available (after the migration is complete). This means that no fallback reading to the immutable field will be needed.

We should therefore simply read the ruleSource.type field to assert if the rule is prebuilt or not.

The KQL query for retrieving prebuilt rules should be:

alert.attributes.params.ruleSource.type: 'external'

Concretely the areas of the application that need updating are:

• the convertRulesFilterToKQL method: used both server-side and in the frontend
• The Mitre Attack dashboard /rules_coverage_overview endpoint (uses convertRulesFilterToKQL)
• The GET /rules/_rule_management_filters endpoint
• The getRuleStatsAggregation method within the Detection Engine health API
• The getExistingPrepackagedRules and getNonPackagedRules methods, used in:
  • The legacy GET /rules/prepackaged/_status endpoint
  • The legacy PUT /rules/prepackaged endpoint
  • The API-only POST /rules/_export endpoint
  • The five new prebuilt rules management endpoints, via IPrebuiltRuleObjectsClient:
    • GET /prebuilt_rules/status
    • POST /prebuilt_rules/installation/_review
    • POST /prebuilt_rules/installation/_perform
    • POST /prebuilt_rules/upgrade/_review
    • POST /prebuilt_rules/upgrade/_perform

Additionally, there's existing telemetry code in the x-pack/plugins/security_solution/server/usage folder that currently relies on the immutable prop and needs to be updated:

• The _getEventLogByTypeAndStatus method
• The getRuleObjectCorrelations method

Backward compatibility

The immutable field is currently accessible through the find endpoint. Once we begin transitioning to rule_source, queries like /api/detection_engine/rules/_find?filter=alert.attributes.params.immutable:false will no longer work. We have to maintain support for the alert.attributes.params.immutable field as the input filter argument for the endpoint. This will involve adding server-side transformations when we receive alert.attributes.params.immutable or alert.attributes.params.ruleSource.type in arguments.

On the front-end, we will continue using the old alert.attributes.params.immutable field to minimize the exposure of our internal data model to the client application.

Background

https://github.com/elastic/kibana/blob/07706fd8e5b423e7e9f4eae096fae24d93d64c2c/x-pack/plugins/security_solution/docs/rfcs/detection_response/prebuilt_rules_customization.md?plain=1#L828-L846

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[banderror]

Blocked by https://github.com/elastic/kibana/pull/183603 and our shift to migrating rules using the Model Version API instead of migration-on-write in our endpoints.