search

uken / fluent-plugin-elasticsearch

Apache License 2.0
892 stars 309 forks source link

Data stream templates and ILM #871

Open snemetz opened 3 years ago

snemetz commented 3 years ago

Is your feature request related to a problem? Please describe. I have component templates, index templates, and ILM policies on ES. But during fluentd startup, default index templates and ILM policies are written for data streams. These overwrite any existing templates and ILM. These are written whether data stream already exists or not. This prevents the use of component templates and setups on ES. This is also a different behavior than what is done for indices

Describe the solution you'd like 1 of 2 options.

  1. Do not write templates and ILM unless specified. This is the behavior used for indices. Since data streams require templates, fail if data stream write/create fails
  2. Only write default template and ILM if data stream write and create fail

Describe alternatives you've considered

Since current behavior overwrites on startup. Fluentd can not be in an autoscaling group. But for static servers, I will be testing the following:

  1. Create templates and ILM
  2. Create data stream
  3. Write something to data stream
  4. start fluentd
  5. upload templates and ILM again

This is a bit of a pain. But It might work temporarily

wbrefvem commented 3 years ago

Fluentd can not be used in an autoscaling group

Or, to put it another way, this plugin can't be used in Kubernetes without significant workarounds. If you're running a fluentd daemonset to scrape k8s pod logs, any time a fluentd pod dies -- or in fact any time the fluentd container restarts -- the ILP and index template associated with the data stream will be overwritten. There's currently no way to decouple these.

Looks like there's a simple fix -- just don't update the ILM policy or the index template if the data stream already exists.

cosmo0920 commented 3 years ago

At master has temporary fix but we still should open this issue for final fixes/improvements.

srook commented 5 months ago

Are there any plans on implementing first suggested solution? Parameter to skip creation of template/ILM?

User should be able to decide if templates/ILMs should be managed through Fluent or Elastic, and not be forced to manage them through Fluent.