search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.77k stars 8.17k forks source link

[Security Solution] Gracefully handle `data_stream.*` fields in alert source data #171104

Open ymao1 opened 10 months ago

ymao1 commented 10 months ago

The alerting framework considers the data_stream.* fields to be reserved fields in the alerts as data indices, particularly since AAD in serverless are actually datastreams so we may want to set specific values for these fields in the future.

Currently, data_stream.* fields from the source index are copied over as-is into data_stream.* fields in the alert document. This was causing issues with the default ECS mapping for these fields (which is constant_keyword) so these fields are actually unmapped in the alert document.

To handle these fields more gracefully, we suggest either (1) Copying data_stream.* fields into a different field within the alerts index that security solutions can define a mapping for (similar to how event.* fields are copied into kibana.alert.original_event.*). (2) Stripping these fields altogether when building the alert document. It seems that event.dataset === data_stream.dataset so that information is already captured in the source document in a different field anyway.

Open to other solutions but the end result should be that the alert document built by the detection engine does not contain the data_stream.* fields in their original path.

elasticmachine commented 10 months ago

Pinging @elastic/security-solution (Team: SecuritySolution)

NeilADesai commented 7 months ago

This causes issues when trying to show the power of Elastic and removing a lot of alerts that are not needed to be seen for an investigation or a threat hunt. In the EDEN AWS demo there are a lot of alerts from 'data_stream.dataset : cloud_security_posture.findings' that should be able to be filtered out instead of having to do each alert by name.

ar3diu commented 7 months ago

I'm also interested in this issue. I would like to be able to use data_stream.namespace field in exceptions and alert searches.

ar3diu commented 7 months ago

Any updates on this? :)

elasticmachine commented 6 months ago

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine commented 6 months ago

Pinging @elastic/security-detection-engine (Team:Detection Engine)