search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.75k stars 24.68k forks source link

Allow alias fields to target unavailable fields #104744

Open felixbarny opened 8 months ago

felixbarny commented 8 months ago

To ease the transition between ECS and OpenTelemetry semantic convention (SemConv) mappings, we're planning to store the two different types of data in different data streams. With the help of alias fields, we'd like to make it possible to re-use existing queries that target ECS fields on OTel data streams. Similarly, we want to add SemConv aliases to ECS-based data streams so the queries in our own UIs and in custom dashboards can target SemConv data while still making it possible to incorporate data from ECS-based data streams.

More on how we'd like to map OTel data in this PR: https://github.com/elastic/elasticsearch/pull/104455.

One of the challenges is this requirement of alias fields:

The target field must exist at the time the alias is created.

We'd like to move away from explicitly defining fields in the mapping and instead rely on dynamic templates (such as ecs@mappings, so that the mappings only contain fields that are actually used.

However, this is incompatible with the requirement that the target for alias fields must already exist.

cc @dakrone

elasticsearchmachine commented 8 months ago

Pinging @elastic/es-search (Team:Search)

elasticsearchmachine commented 8 months ago

Pinging @elastic/es-storage-engine (Team:StorageEngine)

javanna commented 8 months ago

This reminds of the need to move alias fields to be implemented as runtime fields, see https://github.com/elastic/elasticsearch/issues/87969 . I don't think we should make further changes to the alias field implementation, as many of the problems we encounter with it, would be solved by re-implementing them as runtime fields. We could take this additional requirement around targeting unavailable fields into account perhaps.

elasticsearchmachine commented 2 months ago

Pinging @elastic/es-search-foundations (Team:Search Foundations)