Future of alerts-as-data

[kobelb]

Goals

• Set us up for long-term success
• Minimal changes to existing approach
• Support a seamless migration path for existing users

Changes

1) All alert indices have the same mappings

• No more security solution or observability specific alert fields
  • Allows alerts to be queried in a uniform manner
  • Allows platform components to be easily reused by solutions
  • Reduced maintenance burden for solution developers and response ops team
• Alert indices will by default have all ECS standard and extended fields in addition to the kibana.* fields

2) All alerts will be written to the same "datastream"

• No more security solution or observability specific datastreams
• Security solution will no longer put the Kibana space ID in the datastream name
  • This transition will be made gradually
  • This will significantly reduce the risk of mappings explosions by decreasing Elasticsearch's data node heap usage

3) All alerts authorization will be the same

• Additional analysis and collaboration is required to determine the specific of the following
• Authorizing users to access alerts will be consistent for all alerts. For example, users will be granted access to read security solution alerts in the same manner that they're granted access to observability alerts

Implementation

Scope

The following phases are focused on stabilizing and standardizing the alert mappings and "datastreams", which will allow the Alerting framework to persist all alerts-as-data documents and enable us to build framework level features that take advantage of these alerts. These phases can be delivered in the short term. A longer term parallel effort will be ongoing to create a unified authorization model for all alerts.

As a result, not all of the changes from the will be implemented with the following phases. Notably, the following short-comings will be present:

• security solution won't be able to write alerts to the .alerts-default indices
• RBAC for security solution alerts will be different than RBAC for stack and observability alerts

Even with these short-comings, these changes will greatly improve the maintainability of alerting features for the response ops and solution teams while allowing us to create additional framework features that work with and benefit from alerts-as-data.

Phases

Phase 1 - Standardize default alert fields (https://github.com/elastic/kibana/issues/140090)

Summary

• Gain consensus on all the fields
• Create and document common schema for all alerts and rule types

The goal of this phase is to standardize the default list of alert fields in preparation for the future phases. All ECS fields will be part of the default alert fields. As a result, this effort will be solely concerned with standardized the use of the ECS fields and standardizing the kibana.* fields. It will require a collaborative effort between the response ops team and the solution teams to come to agreement on the full list of fields that will by default be included in alert documents. In phase 3A, these default alert fields will be supplemented by user-defined custom fields. The output of this phase will be a full schema for the alerts.

We do not anticipate this effort significantly impacting existing consumers of the alerts-as-data indices. However, some due diligence and investigation will be required to ensure that any changes do not require additional mitigations.

Phase 2 - Framework persists alerts-as-data documents

Summary

• .alerts-default index created on startup with the standardized fields from phase 1
• Enhance the alerting framework APIs to allow rule implementations to return all alerts to the framework ¹
• Alerting framework will persist these alerts as alerts-as-data documents
  • alerts that abide by the standards identified in phase 1 will be written to the .alerts-default indices
  • non-standardized alerts will continue to be written to the existing solution-specific indices
• Stack Management alerts UI will list stack alerts ²

Currently, alerting rule implementations must explicitly use the rule_registry plugin to write alerts-as-data documents in addition to returning the legacy alert instances to the alerting framework to trigger actions. This puts a significant unnecessary burden on alerting rule implementers and has led to only some alerting rules creating alerts-as-data documents. This phase will be focused on making it so that all alerting rule implementations will write alerts-as-data documents in a uniform manner while taking advantage of the standardized alert fields defined in phase 1.

The alerting framework will create a new .alerts-default index on startup with the standardized fields created in phase 1. This will allow all alerting rule implementations to return alerts with the standardized fields to the framework, which will then be persisted to the .alerts-default index automatically. Additionally, facilities will be put in place to allow non-standardized alerts to be returned that will be written to an explicit index. This will allow security detection alerts to continue to be written to the existing .alerts-security.alerts-{spaceId} indices while the unified alert authorization project is ongoing and there are still dependencies on the non-standardized "datastream" naming scheme.

_{1. Special considerations may need to be made to maintain security solution's custom action triggering logic 2. Until we resolve authorization discrepancies we won't be able to list all alerts in the same table at the same time}

Phase 3a - User-defined custom alert fields

Summary

• Create UI / API to manage custom fields
• Custom fields will be #mapped in alert documents ¹
• Custom fields will be indexed and behave like default alert fields: searchable, sortable, etc.

To facilitate pivoting and second-order analytics on the generated alerts, we will be adding a UI and HTTP API to allow users to specify ECS-complaint custom mapped alert fields. These custom fields will behave identically to the standardized default alert fields and will be searchable and sortable. Additional analysis will be required to determine the restrictions that must be put in place on the custom alert fields to minimize the conflicts these custom fields may have with future default fields.

_{1. There's the potential for these custom fields to only work with the standardized alert indices, or also with the non-standardized alert indices. This decision is being delayed until a later time to allow us to make a more educated decision.}

Phase 3b - All alerting rule actions have access to alerts

Summary

• Add a new alert or alerts action variable accessible to all actions, includes all alerts-as-data fields ¹
• Deprecate context.alerts action variables

Currently, only a few alerting rules expose the alerts-as-data to their alerting rule actions, and they are forced to do so by adding a custom context.alerts action variable. This creates an inconsistent and frustrating experience for our end users. Now that alerting rules are all returning their alerts to the framework with the alerts-as-data fields, these alerts can be exposed to all alerting rule actions by adding either an alert or alerts action variable.

_{1. There's the potential for the alert action variables to only work with the standardized alert, or also with the non-standardized alerts. This decision is being delayed until a later time to allow us to make a more educated decision.}

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

Issues for Phase 2:

• https://github.com/elastic/kibana/issues/145100
• https://github.com/elastic/kibana/issues/145103
• https://github.com/elastic/kibana/issues/145107
• https://github.com/elastic/kibana/issues/145108
• https://github.com/elastic/kibana/issues/145115