multiple integrations missing specific templates, ingest pipelines for created data_stream

[rsdrakh]

Hi all,

I realized that multiple integrations I use miss both index templates ansd ingest pipelines for all or some of their datasets and their resulting data streams.

This makes the behaviour of the whole operation very inconsistent and should be fixed.

Here is a list of datasets that I could identify, there is probably more:

metrics-elastic_agent.filebeat_input metrics-endpoint.metadata metrics-endpoint.metrics metrics-endpoint.policy metrics-kubernetes.apiserver metrics-kubernetes.container metrics-kubernetes.controllermanager metrics-kubernetes.event metrics-kubernetes.node metrics-kubernetes.pod metrics-kubernetes.proxy metrics-kubernetes.scheduler metrics-kubernetes.state_container metrics-kubernetes.state_cronjob metrics-kubernetes.state_daemonset metrics-kubernetes.state_deployment metrics-kubernetes.state_job metrics-kubernetes.state_node metrics-kubernetes.state_persistentvolume metrics-kubernetes.state_persistentvolumeclaim metrics-kubernetes.state_pod metrics-kubernetes.state_replicaset metrics-kubernetes.state_resourcequota metrics-kubernetes.state_service metrics-kubernetes.state_statefulset metrics-kubernetes.state_storageclass metrics-kubernetes.system metrics-kubernetes.volume metrics-prometheus.collector metrics-prometheus.query metrics-prometheus.remote_write

They rely on the fallback template "metrics", probably because they don't need any ingest pipelines, still, a consistent behaviour would be much appreciated, because for example every specific template defines a "default_pipeline" that includes a call to a custom pipeline (which our customer uses heavily), and the default metrics template does not.

The possible solutions, in my opinion are:

• create templates and pipelines for every dataset
• include a "default_pipeline" for metrics template

[rsdrakh]

I'd like to add the metrics-kafka datasets.

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!