search

zmoog / public-notes

Apache License 2.0
0 stars 1 forks source link

Figure out how to route documents to a different data stream in Elasticsearch #42

Open zmoog opened 11 months ago

zmoog commented 11 months ago

Elastic introduced rhe reroute processor with https://github.com/elastic/elasticsearch/pull/76511 and it is available since 8.8.0.

I want to explore how to use the new Elasticsearch rerouting capabilities to support new use cases and bring more value to end users.

zmoog commented 11 months ago

Basic example of a pipeline that routes document based on Kubernetes labels:

---
description: Pipeline for Kubernetes container logs
processors:
  - reroute:
      dataset:
        - '{{kubernetes.labels.elastic_co/dataset}}'
        - '{{data_stream.dataset}}'
      namespace:
        - '{{kubernetes.labels.elastic_co/namespace}}'
        - '{{data_stream.namespace}}'

The dataset and namespace can be a single value or and array. It will use the first valid value in the list.