`template_file` and `customize_template` ignored for Data Streams

[joelsdc]

Problem

We are trying to create a non-default index template for data streams but the config (template_file, customize_template) is ignored. I'm no ruby expert but I checked the code and I think the index template is hardcoded to a specific format:

https://github.com/uken/fluent-plugin-elasticsearch/blob/28ade35649cff7a2dd49dd550ac6576cffde8af0/lib/fluent/plugin/out_elasticsearch_data_stream.rb#L113-L133

Steps to replicate

Use template_file and customize_template with elasticsearch_data_stream output.

Expected Behavior or What you need to ask

When using:

        data_stream_template_name "cloud-stream"
        use_legacy_template false
        template_overwrite false
        template_file /etc/fluent/config.d/cloud-index-template.json
        customize_template {"<<NAME>>": "logs-template"}

I expected the newly created index template to have the content of my template_file interpolated with my customize_template config.

I got the default index template created based on the code I linked above.

Using Fluentd and ES plugin versions

• OS version: Debian Bullseye
• Platform? Docker in GKE
• Fluentd: 1.16.2
• ES plugin: 5.2.5
• ES version (optional): 7.16.3

Fix / Notes

Given the code this might not be a bug and is be design, if that's the case, can you please extend this functionality from the elasticsearch output to the elasticsearch_data_stream output?

Thanks!

[bgruszka]

I have the same issue - customize_template doesn't work in elasticsearch_data_stream.

[cosmo0920]

@kenhys @daipom Do you guys have a cycle to handle on this?

[LutzBuchta]

Has there been any update on this ? I have a similar problem with data_stream_template_name

data_stream_template_name logging_template
template_name logging_template
template_file /fluentd/etc/logging_template
template_overwrite true

The Template logging_template is created but with default values. I specified it with template_file but it doesn't take any effect.