search

uken / fluent-plugin-elasticsearch

Apache License 2.0
891 stars 310 forks source link

Plugin doesn't honor logstash_format=false for ILM indices #685

Closed nerzhul closed 4 years ago

nerzhul commented 4 years ago

(check apply)

Problem

The plugin tries to apply logstash default name pattern on index whereas i doesn't want because i'm enabling ILM

2019-12-16 11:07:31 +0000 [error]: #0 Error while index creation - <logstash-default-{now/d}-000001>: #<Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[esc1-4q8h][10.150.240.86:9300][indices:admin/create]"}],"type":"illegal_argument_exception","reason":"The [default] mapping cannot be updated on index [logstash-default-2019.12.16-000001]: defaults mappings are not useful anymore now that indices can have at most one type."},"status":400}>
2019-12-16 11:07:31 +0000 [warn]: #0 failed to flush the buffer. retry_time=5 next_retry_seconds=2019-12-16 11:07:47 +0000 chunk="599d0363501b17c5fbdff7a4e69a7d94" error_class=Elasticsearch::Transport::Transport::Errors::NotFound error="[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [logstash-default-2019.12.16-000001]\",\"index_uuid\":\"_na_\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"logstash-default-2019.12.16-000001\",\"index\":\"logstash-default-2019.12.16-000001\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [logstash-default-2019.12.16-000001]\",\"index_uuid\":\"_na_\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"logstash-default-2019.12.16-000001\",\"index\":\"logstash-default-2019.12.16-000001\"},\"status\":404}"

I have same problem if i remove the 2 lines with logstash_* keys

Steps to replicate

Here is the configuration

<match file/**>
      @type copy
      <store>
        @type elasticsearch
        host {{ .Values.elastic.host }}
        port {{ .Values.elastic.port }}
        include_tag_key true
        tag_key @log_name
        flush_interval 10s
        logstash_format false
        logstash_prefix k8s
        index_name k8s
        rollover_index true
        deflector_alias k8s
        enable_ilm true
        ilm_policy_id k8s
        template_name fluentd
        template_file /fluentd/etc/es-template.json
      </store>
    </match>

Expected Behavior or What you need to ask

...

Using Fluentd and ES plugin versions

cosmo0920 commented 4 years ago

What the name of target rollover index name?

ILM index handling is not same for logstash_format=true mechanism. If you want to stop using datetime format for ILM index, please specify index_date_pattern as "" empty string.

nerzhul commented 4 years ago

hello, i may be wrong i my configuration. Here is the ILM policy:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "90d",
            "max_size": "30gb"
          }
        }
      }
    }
  }
}

do i missed something ?

nerzhul commented 4 years ago

@cosmo0920 , adding index_date_pattern "" enhance the behaviour thanks, but it doesn't work either. Now we dropped the date, nice, but why it wants logstash-default always ?

2019-12-18 08:58:12 +0000 [warn]: #0 failed to flush the buffer. retry_time=7 next_retry_seconds=2019-12-18 08:59:21 +0000 chunk="599f6a0cf0be5c223007b7a48132bff1" error_class=Elasticsearch::Transport::Transport::Errors::NotFound error="[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [logstash-default-000001]\",\"index_uuid\":\"_na_\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"logstash-default-000001\",\"index\":\"logstash-default-000001\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [logstash-default-000001]\",\"index_uuid\":\"_na_\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"logstash-default-000001\",\"index\":\"logstash-default-000001\"},\"status\":404}"
nerzhul commented 4 years ago

i also have a template as described in ES documentation:

{
      "order": 2,
      "index_patterns": ["k8s-*"],
      "settings": {
        "number_of_shards": 3,
        "index.lifecycle.name": "k8s",
        "index.lifecycle.rollover_alias": "k8s"
      },
      "mappings": {}
}
nerzhul commented 4 years ago

@cosmo0920 found the issue, it's a bug in your logging in fluentd. I created the init indice for the ILM policy and now it works as intended :) it seems you don't honor the index_name in the logging part. What i propose as enhancements for the plugin:

cosmo0920 commented 4 years ago

701 should fix this issue. If any issue or question, feel free to comment on this. Closing.