[Security Solution][Detections][Meta] Modularize the Detection Engine

[spong]

This is the meta ticket for tracking the modularization of the Detection Engine. The below is our first steps in supporting RAC (Rules/Alerts/Cases) everywhere, and all efforts are still open for discussion. 🙂

High level feature-sets

Exceptions

Within the main executor, the exceptions logic can be specific to certain rule types (e.g. createThreatSignals() & buildEqlSearchRequest()), added generically as an esFilter pre-query (threshold rules), or applied as a post-filter (e.g.filterEventsAgainstList() for ML rules).

Alert De-duplication

The alert de-duplication logic currently lives within single_bulk_create, and signal_rule_alert_type for EQL rules.

Gap Detection Remediation

Lives within signal_rule_alert_type and is injected into each rule type logic so they can perform the desired searches over the calculated gaps.

Monitoring Efforts

Removal of side-car SO for Rule Execution monitoring in favor of leveraging the Alerting Event Log https://github.com/elastic/kibana/pull/94143

---

Task Breakdown

• [x] Move exceptions builder to lists plugin https://github.com/elastic/kibana/pull/94002 (@marshallmain )
• [x] Split executors for each rule type into separate functions - should include adding rule-type specific validation of params https://github.com/elastic/kibana/pull/94857
  • LOE: medium // Merge conflict potential: high
• [x] Split functions like bulkCreate and searchAfter into smaller, composable functions (@ecezalp ) https://github.com/elastic/kibana/pull/99849
  • LOE: medium // Merge conflict potential: medium
• [x] Extract multiple timestamps logic into separate function (@dhurley14 ) https://github.com/elastic/kibana/pull/96078
  • LOE: medium // Merge conflict potential: low
• [ ] Rename/refactor types, especially SearchResponse types
  • LOE: high // Merge conflict potential: high
• [ ] Remove near-duplicate functions (buildRule, bulkCreate) (TBD)
  • LOE: low // Merge conflict potential: low
• [ ] Move exceptions to lists plugin (@marshallmain )
  • LOE: low // Merge conflict potential: low
• [ ] Update lists plugin to use new ES client (TBD)
  • LOE: ?? // Merge conflict potential: low
• [x] Migrate Exception List UI's (@yctercero) (PR1) (PR2) (PR3)
  • LOE: medium // Merge conflict potential: low
• [ ] Rework Status/Monitoring https://github.com/elastic/kibana/pull/94143 (@banderror)
  • LOE: high // Merge conflict potential: medium/high
• [x] Remove rule executor dependencies on RuleStatusService
  • EQL, ML, Threshold have dependency
  • warnings and errors can be returned from the executor functions and handled at the top level signalRuleAlertType. rule registry can use the new event log service to handle the returned warnings and errors
• [x] Move gap remediation logic out of searchAfterAndBulkCreate (issue describing bug around maxSignals and gap detection with threat match rules)

Potential additional efforts:

• ...
• [ ] Move eventsTelemetry logic out of searchAfterBulkCreate and up to top level so we get telemetry for all rule types (low priority)

Reference docs (internal):

• Rule executor cleanup (slides)
• Detection Engine enhancements

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[marshallmain]

Next refactoring steps:

• finish work on https://github.com/elastic/kibana/pull/99849
• Allow buildSignalGroupFromSequence and buildSignalFromEvent to be passed in to EQL executor function in preparation for sharing with rule registry EQL implementation
  • buildSignalFromEvent and buildBulkBody are almost identical, can be merged into one function as part of the Remove near-duplicate functions (buildRule, bulkCreate) (TBD) task
• Move gap remediation logic out of searchAfterAndBulkCreate (issue describing bug around maxSignals and gap detection with threat match rules)
• Remove dependencies on signalRuleAlertType specific utilities
  • RuleStatusService
  • EQL, ML, Threshold have dependency
  • warnings and errors can be returned from the executor functions and handled at the top level signalRuleAlertType. rule registry can use the new event log service to handle the returned warnings and errors
  • Any other dependencies that won't exist in rule registry, buildRuleMessage maybe?
• Move eventsTelemetry logic out of searchAfterBulkCreate and up to top level so we get telemetry for all rule types? I haven't worked with this before so not 100% clear on this.

[marshallmain]

Additional step:

• Threshold rules depend on querying existing threshold signals to do duplicate mitigation, and they use ruleParams.outputIndex as the index to query. This index will no longer exist in the RAC implementation, so we'll need to conditionally switch to querying the .alerts index and processing alerts in that index appropriately.

[madirey]

Gap remediation working branch: https://github.com/elastic/kibana/compare/master...madirey:rac-gap-remediation?expand=1

[madirey]

Next steps:

• [x] Component template index mapping upgrades (PR)
• [ ] Rule data client changes / consolidation @banderror @xcrzx
  • [ ] rule_registry bootstrapping issues / race conditions
  • [ ] enforce documents aren't written until component templates are applied
  • [ ] error handling? if index bootstrapping fails, we should not be able to write to indices (otherwise we end up with
  • [ ] allow template to be passed in to service to receive scoped client instance (issue)
  • [ ] Converge RuleDataClient and AlertDataClient to fetch and update alerts? / sync with RBAC dynamic mappings)
• [x] Complete a reference rule implementation end-to-end (Custom Query) (PR) @madirey
  • [x] Write BaseSecurity rule type that will do
    • [x] gap detection
    • [x] privilege checks
    • [x] fetch exceptions
    • [x] fetch list client
    • [x] type safety
    • [x] RAC implementations of wrapHits - new schema, wrapSequenceHits - new schema, bulkCreate - ruleDataClient)
  • [x] Map existing fields onto RAC Rule schema / handle eventCategoryOverride and timestampOverride @madirey
  • [x] RAC exceptions? BaseRuleType fetches exceptions and passes them into executors @madirey
  • [x] Migrations
    • [x] rule types
    • [ ] rule statuses? (TODO: maybe acceptable to drop this data)
  • [x] Modify the Threshold Rule type to keep existing alerts state in alerts instance instead of querying alerts index. @madirey
  • [x] Fix threshold cardinality bug #95258 @madirey
  • [ ] Update Gap detection logic in Indicator Match rule (Deprioritized in 7.15 CTI planning)
  • [ ] Handle eventsTelemetry for Custom Query rule type
• [x] Research backwards compatibility with .siem-signals index https://github.com/elastic/kibana/issues/100103 @marshallmain PR
  • [ ] Do we rely on _source in any critical areas in the app? ((1) signals on signals / dupes, (2) Cases - mustache templating, (3) rule actions? -- edge case, but what's the impact?)
• [x] Rule execution log implementation (only use system indices, users can't build visualizations/dashboards) @xcrzx
  • [x] Integrate with BaseSecurity rule type. Pass in generic status writer. Executors call generic status writer to write warning and errors during execution.
• [ ] RBAC: Observability alerts on alerts concerns