Open marshallmain opened 2 years ago
Pinging @elastic/security-detections-response (Team:Detections and Resp)
Pinging @elastic/security-solution (Team: SecuritySolution)
We're looking at resolving this for 8.4 now. The 2 options we have considered are:
data_stream.*
fields as keyword
to alerts indices. This deviates from the ECS standard slightly and also means that the fields will not conform to the data stream naming scheme in which the data_stream
fields are supposed to correspond to the actual name of the ES data stream the document lives in. However, the fields will be queryable in the same ways and with the same queries that customers use on their source data.data_stream.*
fields in a nested location to avoid any possibility of conflicting with ECS or actual data stream usages. The fields might be something like kibana.data_stream.*
. However, user queries will need to account for the different location of this field since we'd be copying data_stream.*
fields in source documents to kibana.data_stream.*
in alerts.Re: option 1, @kobelb mentioned that more of ECS may become a first-class concept in the future. If we populate data_stream.*
fields in non-data stream indices (the alerts indices) and map them as keyword
instead of constant_keyword
, could that cause conflicts with future ECS related features, e.g. enforcing the data stream naming scheme? @jpountz @javanna
In addition, the Security alerts indices use large portions of the ECS mappings. The solutions in https://github.com/elastic/ecs/issues/1869 sound like they could reduce the burden (both in code and in the cluster state) of maintaining those ECS mappings by building them in to ES. If we map these fields as keyword
now, would that be likely to prevent us from adopting ES built-in ECS mappings?
cc @elastic/response-ops @elastic/observability @MikePaquette @jethr0null
Drive-by here, but it seems to me that data_stream
for a given data stream (index, mapping, etc) should stay consistent with the data stream that's storing the documents.
What https://github.com/elastic/kibana/issues/123420 seems to be highlighting is a need for something like (source).data_stream.*
which could deviate from the name of the destination data stream for the document.
Since source
is already used in a network context, it might make sense to introduce to introduce another field set (like option 2).
Naming is hard but maybe derivative
or from
which is a field set expected to contain a copy of an original ECS document from another data stream?
@marshallmain for future reference, you'll probably want to tag @elastic/observability-ui
rather than @elastic/observability
.
bumping it, a customer is looking for a fix for it.
bumping it, a customer is looking for a fix for it.
+1
Any news? Apologies for bumping it again.
Is this still being tracked and worked on? We've implemented a workaround with runtime fields but this is still not perfect.
We would love this working also. We have the biggest issues where we have a single cluster SIEM polling multiple other cluster alert indexes over CCS.
@elastic/response-ops-ram would it be ok to move this ticket over to your team?
In https://github.com/elastic/kibana/pull/123486 we removed the
constant_keyword
data stream fields to avoid situations where alerts became un-indexable if they had varying values for those fields. However, we didn't add regularkeyword
field mappings to replace them similar to how we did for the legacy.siem-signals
mappings. The result is that queries ondata_stream.*
fields don't work for alerts in 8.0+.