search

uken / fluent-plugin-elasticsearch

Apache License 2.0
891 stars 310 forks source link

ES Data Streams: "_policy" appended twice for dynamic stream names (potentially for statics too) #929

Closed applike-ss closed 2 years ago

applike-ss commented 2 years ago

Problem

I am using the elasticsearch_data_stream output with dynamic data_stream_name based on the tag. It seems that the policy generated has "_policy" twice in the name.

So while i would assume this:

i do get this:

I did have a quick look into the code and assume that it relates to data_stream_ilm_name being appended with _policy in multiple occurences: https://github.com/uken/fluent-plugin-elasticsearch/blob/master/lib/fluent/plugin/out_elasticsearch_data_stream.rb#L30 and https://github.com/uken/fluent-plugin-elasticsearch/blob/master/lib/fluent/plugin/out_elasticsearch_data_stream.rb#L82

While my finding could be true and a solution could be as simple as removing the appending of _policy in the create_ilm_policy function, i am not enough into ruby (first day today) that i feel confident of not breaking something.

Steps to replicate

config:

            <source>
              @type forward
              port 15000
            </source>

            <filter **>
                @type record_transformer
                <record>
                    fluentd_hostname "#{Socket.gethostname}"
                    tag ${tag}
                </record>
            </filter>

            <match **>
              @type copy

              <store>
                <buffer tag>
                  @type file
                  chunk_limit_size 5M
                  flush_at_shutdown true
                  flush_interval 5s
                  flush_mode interval
                  flush_thread_count 8
                  overflow_action drop_oldest_chunk
                  path /fluentd/log/elastic-buffer
                  retry_max_interval 30
                  retry_max_times 100
                  retry_timeout 1h
                  total_limit_size 512M
                </buffer>
                @type elasticsearch_data_stream
                host "elasticsearch-master"
                port 9200
                data_stream_name logs-${tag}
                data_stream_template_name logs-${tag}
                reload_connections false
                reconnect_on_error true
                reload_on_failure true
                flush_mode interval
                flush_interval 5s
                suppress_type_name true
              </store>

              ......

            </match>

...

Using Fluentd and ES plugin versions

Fluentd boot logs:

2021-11-15 13:19:39 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/../../etc/fluent/fluent.conf"
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.2'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.16'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.8.1'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.2'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-11-15 13:19:39 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2021-11-15 13:19:39 +0000 [info]: gem 'fluentd' version '1.14.2'
2021-11-15 13:19:39 +0000 [debug]: 'host elasticsearch-master' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-master' doesn't have tag placeholder
2021-11-15 13:19:39 +0000 [debug]: 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2021-11-15 13:19:39 +0000 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2021-11-15 13:19:39 +0000 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2021-11-15 13:19:39 +0000 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2021-11-15 13:19:39 +0000 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2021-11-15 13:19:39 +0000 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2021-11-15 13:19:39 +0000 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2021-11-15 13:19:39 +0000 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2021-11-15 13:19:39 +0000 [debug]: Need substitution: false
2021-11-15 13:19:39 +0000 [debug]: 'host_placeholder elasticsearch-master' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-master' doesn't have tag placeholder
2021-11-15 13:19:39 +0000 [info]: using configuration file: <ROOT>
fiscafusca commented 2 years ago

Hi @applike-ss, you are correct! You can go ahead and remove "_policy" from this occurrence: https://github.com/uken/fluent-plugin-elasticsearch/blob/master/lib/fluent/plugin/out_elasticsearch_data_stream.rb#L82

Thank you for pointing it out! :)

applike-ss commented 2 years ago

@fiscafusca thank you, i will prepare a PR. Currently checking and fixing tests.