search

uken / fluent-plugin-elasticsearch

Apache License 2.0
891 stars 309 forks source link

Can't configure dynamic index / templates with ILM #867

Closed aalemanq closed 3 years ago

aalemanq commented 3 years ago

(check apply)

Problem

Can't configure ILM with dynamic template creation for every index with $tag created.

Steps to replicate

Either clone and modify https://gist.github.com/pitr/9a518e840db58f435911

OR

With this config I can't autogenerate templates for every index-$tag, it is posible? I'm trying to apply this without success : https://github.com/uken/fluent-plugin-elasticsearch/pull/708

<match docker.gitlab.**>
    @type elasticsearch_dynamic
    suppress_type_name true
    type_name "_doc"

    index_name               xxx-test-${tag_parts[1]}
    time_key                 time
    include_timestamp        true
    include_tag_key          true
    flatten_hashes           true
    flatten_hashes_separator _
    rollover_index     true
    application_name   ${tag_parts[1]}
    index_date_pattern ""
    deflector_alias    fluentd-${tag_parts[1]}

    # Index template
    template_name      fluentd-${tag_parts[1]}
    template_file      /fluentd/etc/mappings/xxx-test.mapping
    customize_template {"<<TAG>>":"${tag_parts[1]}"}
    template_overwrite true

    hosts xxx
    user fluentd
    password xxx
    include_tag_key true
    tag_key "original_tag"
    #You can tune how the elasticsearch-transport host reloading feature works. By default it will reload the host list from the server every 10,000th request to spread the load.defaults to true
    reload_connections true
    #Indicates that the elasticsearch-transport will try to reload the nodes addresses if there is a failure while making the request. defaults false
    reload_on_failure true
    #You can set in the elasticsearch-transport how often dead connections from the elasticsearch-transport's pool will be resurrected. defaults to 60s
    resurrect_after 30
    reconnect_on_error true
    <buffer>
      flush_thread_count 8
      flush_interval 1s
      chunk_limit_size 10M
      queue_limit_length 16
      retry_max_interval 30
      retry_forever true
    </buffer>
</match>
{
  "order": 2,
  "index_patterns": [
    "xxx-test-<<TAG>>*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "3daysToSata",
    "rollover_alias": "xxx-test-<<TAG>>"
      },
      "codec": "best_compression",
      "routing": {
        "allocation": {
          "require": {
            "box_type": "ssd"
          }
        }
      },
      "refresh_interval": "5s",
      "number_of_shards": "2",
      "number_of_replicas": "2"
    }
  },
  "mappings": {},
  "aliases": {}
}

Expected Behavior or What you need to ask

Create 1 template for every index with -$tag for example: --xxx-test-app1 with template xxx-test-app1 and rollooveralias xxx-test-app1 --xxx-test-app2 with template xxx-test-app2 and rollooveralias xxx-test-app2 --xxx-test-app3 with template xxx-test-app3 and rollooveralias xxx-test-app3

Using Fluentd and ES plugin versions

LOCAL GEMS

async (1.28.7) async-http (0.46.3) async-io (1.30.2) bigdecimal (1.3.5) cmath (default: 1.0.0) console (1.10.1) cool.io (1.7.0) csv (default: 1.0.0) date (default: 1.0.0) dig_rb (1.0.1) elasticsearch (7.10.1) elasticsearch-api (7.10.1) elasticsearch-transport (7.10.1) elasticsearch-xpack (7.10.1) etc (default: 1.0.0) excon (0.79.0) faraday (1.3.0) faraday-net_http (1.0.1) fcntl (default: 1.0.0) fiber-local (1.0.0) fileutils (default: 1.0.2) fluent-config-regexp-type (1.0.0) fluent-plugin-elasticsearch (5.0.0) fluent-plugin-geoip (1.3.2) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-route (1.0.0) fluentd (1.6.0) geoip-c (0.9.1) geoip2_c (0.3.3) http_parser.rb (0.6.0) ipaddr (default: 1.2.0) json (2.2.0) msgpack (1.4.2) multi_json (1.15.0) multipart-post (2.1.1) nio4r (2.5.5) oj (3.3.10) openssl (default: 2.1.2) protocol-hpack (1.4.2) protocol-http (0.8.1) protocol-http1 (0.8.3) protocol-http2 (0.9.7) psych (default: 3.0.2) ruby2_keywords (0.0.4) scanf (default: 1.0.0) serverengine (2.2.2) sigdump (0.2.4) stringio (default: 0.0.1) strptime (0.2.5) strscan (default: 1.0.0) thread_safe (0.3.6) timers (4.3.2) tzinfo (1.2.9) tzinfo-data (1.2021.1) webrick (default: 1.4.2) yajl-ruby (1.4.1) zlib (default: 1.0.0)

* ES version (optional)
7.2.0
* ES template(s) (optional)

ERROR LOGS:

efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [info]: adding match pattern="docker.gitlab.**" type="elasticsearch_dynamic" efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'index_name xxx-test-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: xxx-test-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'template_name fluentd-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: fluentd-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'customize_template ${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'customize_template: ${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'deflector_alias fluentd-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: fluentd-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'application_name ${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: ${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 Need substitution: false efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'index_name xxx-test-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: xxx-test-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'template_name fluentd-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: fluentd-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'customize_template ${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'customize_template: ${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'deflector_alias fluentd-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: fluentd-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'application_name ${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: ${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [debug]: #0 Need substitution: false efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:05 +0100 [info]: #0 Template 'fluentd-${tag_parts[1]}' overwritten with /fluentd/etc/mappings/xxx-test.mapping. efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:07 +0100 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. [403] efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:07 +0100 [warn]: #0 Remaining retry: 9. Retry to communicate after 2 second(s). efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:09 +0100 [info]: #0 Template 'fluentd-${tag_parts[1]}' overwritten with /fluentd/etc/mappings/xxx-test.mapping. efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:13 +0100 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. [403] efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:13 +0100 [warn]: #0 Remaining retry: 8. Retry to communicate after 4 second(s). efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:13 +0100 [info]: #0 Template 'fluentd-${tag_parts[1]}' overwritten with /fluentd/etc/mappings/xxx-test.mapping. efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:21 +0100 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. [403] efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:21 +0100 [warn]: #0 Remaining retry: 7. Retry to communicate after 8 second(s). efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:22 +0100 [info]: #0 Template 'fluentd-${tag_parts[1]}' overwritten with /fluentd/etc/mappings/xxx-test.mapping. efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:38 +0100 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. [403] efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:38 +0100 [warn]: #0 Remaining retry: 6. Retry to communicate after 16 second(s). efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:39 +0100 [info]: #0 Template 'fluentd-${tag_parts[1]}' overwritten with /fluentd/etc/mappings/xxx-test.mapping. efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:52 +0100 [info]: Received reload efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: Worker 0 finished unexpectedly with signal SIGUSR2 efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.0' efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: gem 'fluent-plugin-geoip' version '1.3.2' efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0' efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: gem 'fluent-plugin-route' version '1.0.0' efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: gem 'fluentd' version '1.6.0' efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: adding match pattern="syslog.kern.info" type="route" efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: adding filter pattern="iptables.kern.info" type="parser" efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: adding match pattern="noiptables.kern.info" type="null" efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: adding match pattern="fluent.*" type="null" efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:53 +0100 [info]: adding match pattern="iptables.kern.info" type="elasticsearch" efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:54 +0100 [info]: adding match pattern="{stdout.docker.json.geoip.-proxy.,stdout.docker.json.geoip.prod-proxy.}" type="elasticsearch" efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [info]: adding match pattern="docker.gitlab.**" type="elasticsearch_dynamic" efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'index_name xxx-test-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: xxx-test-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'template_name fluentd-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: fluentd-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'customize_template ${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'customize_template: ${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'deflector_alias fluentd-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: fluentd-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'application_name ${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: ${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 Need substitution: false efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'host_placeholder localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: localhost' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'host localhost' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: localhost' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'index_name xxx-test-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: xxx-test-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'template_name fluentd-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: fluentd-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'customize_template ${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'customize_template: ${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'deflector_alias fluentd-${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: fluentd-${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'application_name ${tag_parts[1]}' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: ${tag_parts[1]}' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:55 +0100 [debug]: #0 Need substitution: false efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:56 +0100 [info]: #0 Template 'fluentd-${tag_parts[1]}' overwritten with /fluentd/etc/mappings/xxx-test.mapping. efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:56 +0100 [error]: #0 Error while index creation - <xxx-test-${tag_parts[1]}-${tag_parts[1]}-000001> error_class=Elasticsearch::Transport::Transport::Errors::Forbidden error="[403] {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]: [failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:admin/create] is unauthorized for user [fluentd]\",\"caused_by\":{\"type\":\"parse_exception\",\"reason\":\"failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]: [failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Text 'tag_parts[1]' could not be parsed at index 0\"}}}},\"status\":403}" efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:58 +0100 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. [403] {"error":{"root_cause":[{"type":"parse_exception","reason":"failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]: [failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]]"}],"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [fluentd]","caused_by":{"type":"parse_exception","reason":"failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]: [failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]]","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]","caused_by":{"type":"date_time_parse_exception","reason":"Text 'tag_parts[1]' could not be parsed at index 0"}}}},"status":403} efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:58 +0100 [warn]: #0 Remaining retry: 9. Retry to communicate after 2 second(s). efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:59 +0100 [info]: #0 Template 'fluentd-${tag_parts[1]}' overwritten with /fluentd/etc/mappings/xxx-test.mapping. efk-prod_aggregator.1.alrrsak27xff@devops-worker12 | 2021-02-10 13:22:59 +0100 [error]: #0 Error while index creation - <xxx-test-${tag_parts[1]}-${tag_parts[1]}-000001> error_class=Elasticsearch::Transport::Transport::Errors::Forbidden error="[403] {\"error\":{\"root_cause\":[{\"type\":\"parse_exception\",\"reason\":\"failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]: [failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:admin/create] is unauthorized for user [fluentd]\",\"caused_by\":{\"type\":\"parse_exception\",\"reason\":\"failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]: [failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"failed to parse date field [tag_parts[1]] with format [uuuu.MM.dd]\",\"caused_by\":{\"type\":\"date_time_parse_exception\",\"reason\":\"Text 'tag_parts[1]' could not be parsed at index 0\"}}}},\"status\":403}"

aalemanq commented 3 years ago

Has anybody set up mappings in base enviroments with ILM included?

aalemanq commented 3 years ago

Y tried adding buffer settings but same error about date and logstash...:

 <buffer tag,time>
      @type file
      timekey 1h
      path /fluentd/log/es-out

      flush_thread_count 4
      flush_interval     5s
      flush_at_shutdown  true

      chunk_limit_size   2M
      total_limit_size   256M

      retry_max_interval 30
      retry_forever      true
    </buffer>
</match>
Asgoret commented 3 years ago

Familiar problem. Can't customize index template

Asgoret commented 3 years ago

Code:

{
  "index_patterns": ["mock"],
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "mock",
          "rollover_alias": "mock"
        },
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    }
  }
}
customize_template {"number_of_replicas":"2"}
Asgoret commented 3 years ago

@aalemanq for ILM (my config. It created and updates). indexes also created, but I can't customize the index template. @cosmo0920 can you help us plz?

    # ILM Settings - WITH ROLLOVER support
    # https://github.com/uken/fluent-plugin-elasticsearch#enable-index-lifecycle-management
    rollover_index true
    application_name "etl-webserver"
    index_date_pattern ""
    # Policy configurations
    enable_ilm true
    ilm_policy_id etl-webserver
    ilm_policy_overwrite true
    ilm_policy {"policy": {"phases": {"hot": {"min_age": "0ms","actions": {"rollover": {"max_age": "5m","max_size": "3gb"},"set_priority": {"priority": 100}}},"delete": {"min_age": "30d","actions": {"delete": {"delete_searchable_snapshot": true}}}}}}
    use_legacy_template false
    template_name etl-webserver
    template_file /configs/index-template.json
    template_overwrite true
    customize_template {"<<number_of_replicas>>":"1"}
Asgoret commented 3 years ago

Ok, I resolved all my issues. Can answer some questions @aalemanq

aalemanq commented 3 years ago

Wow! Thanks god!!

@Asgoret just...can you share with me your last full config that is works with template content too (with bufer etc..)? I can't generate templates "sed" is not working ...:( and I have errors about logstash timestamp data....

Asgoret commented 3 years ago

@aalemanq sure!

es index template:

{
  "index_patterns": ["mock"],
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "mock",
          "rollover_alias": "mock"
        },
        "number_of_shards": "<<shard>>",
        "number_of_replicas": "<<replica>>"
      }
    }
  }
}

fluentd config

<source>
  @type http
  port 5004
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
  <parse>
    @type json
  </parse>
</source>

<match kubernetes.var.log.containers.**etl-webserver**.log>
    @type elasticsearch
    @id out_es_etl_webserver
    @log_level info
    include_tag_key true
    host $HOST
    port $PORT
    path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}"
    request_timeout "#{ENV['FLUENT_ELASTICSEARCH_REQUEST_TIMEOUT'] || '30s'}"
    scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
    ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
    ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1'}"
    reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}"   
    reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}"
    reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}"
    log_es_400_reason "#{ENV['FLUENT_ELASTICSEARCH_LOG_ES_400_REASON'] || 'false'}"
    logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'etl-webserver'}"
    logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'false'}"
    index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'etl-webserver'}"
    type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}"
    time_key "#{ENV['FLUENT_ELASTICSEARCH_TIME_KEY'] || '@timestamp'}"
    include_timestamp "#{ENV['FLUENT_ELASTICSEARCH_INCLUDE_TIMESTAMP'] || 'true'}"

    # ILM Settings - WITH ROLLOVER support
    # https://github.com/uken/fluent-plugin-elasticsearch#enable-index-lifecycle-management
    rollover_index true
    application_name "etl-webserver"
    index_date_pattern ""
    # Policy configurations
    enable_ilm true
    ilm_policy_id etl-webserver
    ilm_policy_overwrite true
    ilm_policy {"policy": {"phases": {"hot": {"min_age": "0ms","actions": {"rollover": {"max_age": "5m","max_size": "3gb"},"set_priority": {"priority": 100}}},"delete": {"min_age": "30d","actions": {"delete": {"delete_searchable_snapshot": true}}}}}}
    use_legacy_template false
    template_name etl-webserver
    template_file /configs/index-template.json
    template_overwrite true
    customize_template {"<<shard>>": "3","<<replica>>": "0"}

    <buffer>
        flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
        flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
        chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '8MB'}"
        total_limit_size "#{ENV['FLUENT_ELASTICSEARCH_TOTAL_LIMIT_SIZE'] || '450MB'}"
        queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"
        retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '60s'}"
        retry_forever false
    </buffer>
</match>
Asgoret commented 3 years ago

@aalemanq how it's going?

aalemanq commented 3 years ago

I have to test it!! very very thanks really!! I will tell you something

Niwaiwa commented 3 years ago

I used this config which can use index_name or logstash_format

index template

{
  "index_patterns": ["<<TAG>>-log*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "test-policy", 
    "index.lifecycle.rollover_alias": "<<TAG>>-log"
  }
}

fluentd config

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match {nginx**,httpd**}>
    @type elasticsearch
    host 192.168.1.101
    port 9200
    default_elasticsearch_version 7
    verify_es_version_at_startup true
    suppress_type_name true
    include_tag_key true
    include_timestamp true

    # logstash_format true
    # logstash_prefix ${tag}
    # logstash_dateformat ""

    index_name ${tag}

    template_overwrite true
    template_name ""
    template_file /fluentd/etc/test_template.json
    customize_template {"<<TAG>>": "${tag}"}
    rollover_index true

    application_name log
    enable_ilm true
    ilm_policy_id test-policy
    ilm_policy { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_age": "1m", "max_docs": 20, "max_size": "20kb" } } }, "delete": { "min_age": "1h", "actions": { "delete": {} } } } } }
    ilm_policy_overwrite true

    flush_interval 3s
</match>
chenbr2018 commented 3 years ago

image es7.9.0

StepanKuksenko commented 3 years ago

@Asgoret thank you man ! Your solution works like a charm, added it to my bookmarks for future also, good job man!

Asgoret commented 3 years ago

@cosmo0920 I think issue is resolved ;D

cosmo0920 commented 3 years ago

I guess that we can close this issue after adding FAQ for this problem.

StepanKuksenko commented 3 years ago

also it would be great to add example folder with example configuration such as Asgoret's one.

Asgoret commented 3 years ago

@aalemanq Hi, I get an email from github, but can't find your post..... Screenshot from 2021-04-09 19-04-25

cosmo0920 commented 3 years ago

@aalemanq This was caused by using too new elasticsearch gem, I guess. In some version, elasticsearch-ruby might change HTTP method from POST to PUT.

madanrishi commented 2 years ago

@Asgoret @cosmo0920 is there a way to add conditions in match kind of like if-else statements to choose which index-template to use. to give context I want to use 1 template for most of the namespaces but there are some specific namespaces where I want to use different index template.

    rollover_index true
    application_name "default" # default is set even if not specified
    index_date_pattern ""
    enable_ilm true
    ilm_policy_id hot-warm-cold-delete-30d
    <psuedo code >
    if ${$.kubernetes.namespace_name}" == "kube-proxy"
           tempalte_name kube
           template_file path to file
    template_overrite true
    use_legacy_template false

something like this?

cosmo0920 commented 2 years ago

Fluentd is not Logstash. There is no if condition support on configuration. Thanks.

cosmo0920 commented 2 years ago

If you accomplish your goal, it might help to use record_modifier to handle records: https://github.com/repeatedly/fluent-plugin-record-modifier#ruby-code-trick-for-complex-logic