elasticsearch: Add time chunk_keys supplementary description for time placeholder

[cosmo0920]

This issue is reported in https://github.com/uken/fluent-plugin-elasticsearch/issues/481.

Because time placeholder can work with a chunk which contains timekey information. Default buffer configuration does not contain timekey information but tag information.

Default buffer configuration:

<buffer tag>
  @type memory
</buffer>

When using the above default buffer configuration, elasticseatch plugin does not extract time placeholder:

index_name fluentd.${tag}.%Y%m%d -> index_name fluentd.your.tag.%Y%m%d

tag and time in chunk_keys:

<buffer tag, time>
  @type memory
  timekey 1h
</buffer>

When using buffer configuration with timekey, elasticsearch plugin can handle time placeholder:

index_name fluentd.${tag}.%Y%m%d -> index_name fluentd.your.tag.20181011

Signed-off-by: Hiroshi Hatake hatake@clear-code.com

[fujimotos]

Thank you!

[xuejipeng]

@fujimotos if i have buff config like this and how do i add the <buffer tag, time>?

    <match **>
      @id elasticsearch
      @type elasticsearch
      @log_level info
      type_name _doc
      include_tag_key true
      host 172.30.1.37
      port 9200
      #logstash_format true
      index_name ${tag}-%Y%m%d
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 2M
        queue_limit_length 8
        overflow_action block
      </buffer>
    </match>

[fujimotos]

@xuejipeng As far as I can tell, putting tag,time inside the bracket and setting timekey should do the job.

<buffer>
  @type file
  path /var/log/fluentd-buffers/kubernetes.system.buffer
  timekey 1d
  flush_mode interval
  retry_type exponential_backoff
  flush_thread_count 2
  flush_interval 5s
  retry_forever
  retry_max_interval 30
  chunk_limit_size 2M
  queue_limit_length 8
  overflow_action block
</buffer>

Please try to see if it works. If you find something wrong, report the issue to https://github.com/uken/fluent-plugin-elasticsearch.

[xuejipeng]

@fujimotos it's work

<match **>
  @id elasticsearch
  @type elasticsearch
  @log_level info
  type_name _doc
  include_tag_key true
  host xxx
  port 9200
  #logstash_format true
  #logstash_prefix ${tag}
  index_name tag-%Y%m%d
  <buffer tag,time>
    @type file
    timekey 1d
    path /var/log/fluentd-buffers/kubernetes.system.buffer
    flush_mode interval
    retry_type exponential_backoff
    flush_thread_count 2
    flush_interval 5s
    retry_forever
    retry_max_interval 30
    chunk_limit_size 2M
    queue_limit_length 8
    overflow_action block
  </buffer>
</match>

But I found another good way.

<match **>
  @id elasticsearch
  @type elasticsearch
  @log_level info
  type_name _doc
  include_tag_key true
  host xxx
  port 9200
  logstash_format true
  logstash_prefix ${tag}
  # index_name ${tag}-%Y%m%d
  <buffer>
    @type file
    path /var/log/fluentd-buffers/kubernetes.system.buffer
    flush_mode interval
    retry_type exponential_backoff
    flush_thread_count 2
    flush_interval 5s
    retry_forever
    retry_max_interval 30
    chunk_limit_size 2M
    queue_limit_length 8
    overflow_action block
  </buffer>
</match>