search

uken / fluent-plugin-elasticsearch

Apache License 2.0
888 stars 309 forks source link

How to get rid of debug log "Detected ES 8.x or above: document type will not be used" #1024

Open dwidyna opened 1 year ago

dwidyna commented 1 year ago

(check apply)

Problem

I was using fluentd with your plugin sending logs to Elasticsearch 5.6 . Now I am upgrading ES to 8.8. I changed fluent/plugin configuration and mapping templates to get rid of "types" that are no longer present in ES 8.8. I cannot get rid of this log message "[debug]: #0 [match_es_fluent_all_copy_aws] Detected ES 8.x or above: document type will not be used." (I'm getting same message regarding other tags as well).

Steps to replicate

fluent.conf

<system>
  rpc_endpoint 0.0.0.0:24444
  log_level debug
</system>

<source>
  @type forward
  @id source_forward_generic
  port 24224
</source>

<label @FLUENT_LOG>
  <filter fluent.**>
    @type record_transformer
    @id filter_record_transformer_fluent_all
    renew_record true
    enable_ruby true
    <record>
      timestamp ${time.utc.strftime('%Y-%m-%d %H:%M:%S.%6N')}
      severity ${tag_parts.last}
      message ${record.to_json}
      tag_text ${record.dig("tag")}
      tag_keyword ${record.dig("tag")}
    </record>
  </filter>

  <match>
    @type elasticsearch
    @id match_es_fluent_all_copy_aws

    @include _common_elasticsearch.conf

    #below index name is not used:
    index_name fluentd_logs_fallback

    #index data into yearly rotated index:
    logstash_format true
    logstash_prefix fluentd_logs_v_1_0_0
    logstash_prefix_separator _
    logstash_dateformat %Y

    #take time from record itself but don't index @timestamp field:
    time_key timestamp
    time_key_format "%Y-%m-%d %H:%M:%S.%N"
    time_precision 6
    include_timestamp false
    time_key_exclude_timestamp true

    #load specific index template:
    templates {
      "template_fluentd_logs_v_1_0_0": "/fluentd/etc/es_mappings/template_fluentd_logs.json"
    }
  </match>
</label>

_common_elasticsearch.conf

    #general settings:
    scheme http
    host elasticsearch
    port 9200
    user elasticsearch
    password es_password

    #Managed Elasticsearch service in AWS needs it false (shield case)
    reload_connections true

    validate_client_version true
    verify_es_version_at_startup true
    suppress_type_name true

    max_retry_putting_template 100

    <buffer>
      flush_interval 1s
    </buffer>

/fluentd/etc/es_mappings/template_fluentd_logs.json

{
    "order": 0,
    "index_patterns": [
        "fluentd_logs_v_1_0_0_*"
    ],
    "settings": {
        "index": {
            "number_of_shards": "1",
            "number_of_replicas": "0"
        }
    },
    "mappings": {
        "dynamic": "strict",
        "properties": {
            "timestamp": {
                "format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
                "type": "date"
            },
            "severity": {
                "type": "keyword"
            },
            "tag_keyword": {
                "type": "keyword"
            },
            "tag_text": {
                "type": "text"
            },
            "message": {
                "type": "text"
            }
        }
    },
    "aliases": {}
}

Expected Behavior or What you need to ask

No logs regarding the type as they are polluting logs and I believe no "type" resides in my config.

Using Fluentd and ES plugin versions

dwidyna commented 10 months ago

Hello guys, any chance for solution ?

psandeep09 commented 10 months ago

Hello, any update

Floppe commented 5 months ago

Some months later, any solution?

prateekshetty-eox commented 1 month ago

Any updates here?

harish2497 commented 2 weeks ago

Any updates here?