search

elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
128 stars 138 forks source link

Enhance Agent to also cover non-time-series use-cases #1652

Open dschneiter opened 1 year ago

dschneiter commented 1 year ago

Describe the enhancement: Agent relies on time-series data, ILM and ECS, while Filebeat is also configurable for non-time-series use-cases

Describe a specific use case for the enhancement or feature: On a weekly base, I'm currently uploading weekly exports of training survey responses which also include all historic responses in CSV format to an Elasticcloud deployment using Filebeat. The data is entity-centric with every survey response having a unique ID, does not rely on ECS, and does not require ILM. I was wondering whether I could transition to agent, which currently does not seem to support use-cases like this.

Issue created on request of @ruflin

ruflin commented 1 year ago

Part of the limitation comes from Elastic Agent only allowing shipping data to the data stream naming scheme which is by design. But as data streams prevent to overwrite data, the above use case currently can't be covered.