elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
194 stars 422 forks source link

[Kubernetes] Add top level processor `add_fields` for the cluster name usecase #10878

Open tetianakravchenko opened 3 weeks ago

tetianakravchenko commented 3 weeks ago

One of the common processor that might be used is a add_fields to add a orchestrator.cluster.name field - we mention it in the k8s module doc - https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_dashboard_32:

processors:
  - add_fields:
      target: orchestrator.cluster
      fields:
        name: clusterName
        url: clusterURL

Now we provide an option to set processors on the data_stream level, but it is quite a lot of changes for the such a common scenario.

We should provide a way to set orchestrator.cluster.name on the integration level

tetianakravchenko commented 1 week ago

draft branch https://github.com/elastic/integrations/compare/main...tetianakravchenko:integrations:k8s_add_cluster_name?expand=1 UI: Image

as a result only this processor will be added to all data_stream:

{{#if cluster_name}}
- add_fields:
    target: orchestrator.cluster
    fields:
      name: {{cluster_name}}
{{/if}}
tetianakravchenko commented 1 week ago

Few issues:

  1. In case orchestrator.cluster.name is set - add_fields processor add the value to the array: Image

This behavior actually contradicts the processor description https://www.elastic.co/guide/en/beats/filebeat/current/add-fields.html:

The add_fields processor will overwrite the target field if it already exists.

Issue - https://github.com/elastic/beats/issues/40780

  1. state_namespace metricset does not include orchestrator.cluster.name field in kind env