[exporter/elasticsearch] route based on data stream attributes

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector

https://opentelemetry.io

Apache License 2.0

2.71k stars 2.14k forks source link

[exporter/elasticsearch] route based on data stream attributes #33755

Open andrzej-stencel opened 4 days ago

andrzej-stencel commented 4 days ago

Description:

This adds a feature to the Elasticsearch exporter to route telemetry based on data stream attributes, as described in An introduction to the Elastic data stream naming scheme.

Link to tracking Issue:

Testing:

Added unit tests.

Documentation:

Added documentation in exporter's README.

andrzej-stencel commented 3 days ago

We should ensure that all docs have a data_stream.<type|dataset|namespace> property at the top-level, that's consistent with the _index ("${data_stream.type}-${data_stream.dataset:generic}-${data_stream.namespace:default}" when data_stream mode is enabled.

@felixbarny I have added filling in missing data stream attributes in https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/33755/commits/cb88ebcef50226de5630e4c080f032ce59ef989b, can you take a look?