elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
68.65k stars 24.37k forks source link

Add OpenTelemetry mappings #104731

Open AlexanderWert opened 5 months ago

AlexanderWert commented 5 months ago

To provide first class support for OpenTelemetry, we should work on built-in index templates optimized for OpenTelemetry signals. The goal for these mappings is to be as native to OpenTelemetry/OTLP as possible, even if that means the structure of OTel documents and mappings are different from what they are today.

Yet, we want to provide a compatibility layer so that queries written with ECS and the traditional document structure, where everything is mapped at the top-level of the documents, still work on OTel data to make it possible to gradually migrate to the OTel native data model. See also https://github.com/elastic/elasticsearch/issues/104744

Like we did in https://github.com/elastic/elasticsearch/pull/97546 for APM, we should create an Elasticsearch plugin that ships index templates and component templates with Elasticsearch. The mappings can be based on https://github.com/elastic/elasticsearch/pull/104455.

Let's try to keep the approach similar to apm-data, i.e. minimise the amount of Java code and define templates as YAML like in https://github.com/elastic/elasticsearch/tree/main/x-pack/plugin/apm-data/src/main/resources. We should also have functional tests written as YAML REST tests, like in https://github.com/elastic/elasticsearch/tree/main/x-pack/plugin/apm-data/src/yamlRestTest/resources/rest-api-spec/test.

During development the plugin should be disabled by default, with a setting to enable it. We will enable it by default in serverless QA first to increase confidence, and plan to later enable it by default everywhere.

AlexanderWert commented 5 months ago

PR: https://github.com/elastic/elasticsearch/pull/104455

elasticsearchmachine commented 5 months ago

Pinging @elastic/es-data-management (Team:Data Management)