elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
105 stars 4.92k forks source link

Evaluate how beats can move away of using a single datastream to datastream per module #41702

Open AndersonQ opened 2 hours ago

AndersonQ commented 2 hours ago

Describe the enhancement:

Currently Beats use a single datastream/index template with the mappings for every possible module, making the mappings huge. We've just crossed the line of 10,000 fields in a index with metricbeat 8.15 (see here) had to increased it even further.

A better approach would be to have a dedicated datastream per module, so each datastream/index template would have only the necessary mappings for its module.

This issue aims to evaluate:

Describe a specific use case for the enhancement or feature:

elasticmachine commented 2 hours ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)