uken / fluent-plugin-elasticsearch

Apache License 2.0
891 stars 310 forks source link

Add ILM and template parameters for data streams #920

Closed fiscafusca closed 2 years ago

fiscafusca commented 2 years ago

I added two optional parameters (data_stream_ilm_name and
data_stream_template_name) that can be used to specify the name
of already existing ILM policy and template. If not present in the
configuration, the default policy and template will be created,
as it was before.
The relative test file was edited to work with the new parameters.

When creating a new data stream, one may want to specify an existing
ILM policy and/or index template to be applied to the data stream
(see issue #904). Having pre-made templates with a higher priority is
just a patch that does not solve the root of the problem.
For example, in our application we need to generate multiple data
streams from fluentd. Without this feature, the plugin ends up
generating loads of identical default ILM policies and templates that
remain unused and must be deleted manually.

(check all that apply)

fiscafusca commented 2 years ago

Sorry, I was out of office yesterday. I just checked, it should work now

fiscafusca commented 2 years ago

Fixed a couple of things, the tests do fine now on my machine without changing anything aside the data stream file and its relative test file. Hopefully nothing else should be missing

cosmo0920 commented 2 years ago

Hi @fiscafusca , I sent a PR for following up your work. Ordinary ES plugin users doesn't want to read its implementation for trouble shooting. Invalid Data Stream parameter related error messages should be reported with actual parameter names.

fiscafusca commented 2 years ago

Hi @cosmo0920, just checked your PR! You're right, I definitely missed this. Thank you for following up