Closed cosmo0920 closed 5 years ago
Thank you!
@fujimotos if i have buff config like this and how do i add the <buffer tag, time>
?
<match **>
@id elasticsearch
@type elasticsearch
@log_level info
type_name _doc
include_tag_key true
host 172.30.1.37
port 9200
#logstash_format true
index_name ${tag}-%Y%m%d
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.system.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 2M
queue_limit_length 8
overflow_action block
</buffer>
</match>
@xuejipeng As far as I can tell, putting tag,time
inside the bracket
and setting timekey
should do the job.
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.system.buffer
timekey 1d
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 2M
queue_limit_length 8
overflow_action block
</buffer>
Please try to see if it works. If you find something wrong, report the issue to https://github.com/uken/fluent-plugin-elasticsearch.
@fujimotos it's work
<match **>
@id elasticsearch
@type elasticsearch
@log_level info
type_name _doc
include_tag_key true
host xxx
port 9200
#logstash_format true
#logstash_prefix ${tag}
index_name tag-%Y%m%d
<buffer tag,time>
@type file
timekey 1d
path /var/log/fluentd-buffers/kubernetes.system.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 2M
queue_limit_length 8
overflow_action block
</buffer>
</match>
But I found another good way.
<match **>
@id elasticsearch
@type elasticsearch
@log_level info
type_name _doc
include_tag_key true
host xxx
port 9200
logstash_format true
logstash_prefix ${tag}
# index_name ${tag}-%Y%m%d
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.system.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 2M
queue_limit_length 8
overflow_action block
</buffer>
</match>
This issue is reported in https://github.com/uken/fluent-plugin-elasticsearch/issues/481.
Because time placeholder can work with a chunk which contains timekey information. Default buffer configuration does not contain timekey information but tag information.
Default buffer configuration:
When using the above default buffer configuration, elasticseatch plugin does not extract time placeholder:
index_name fluentd.${tag}.%Y%m%d -> index_name fluentd.your.tag.%Y%m%d
tag and time in chunk_keys:
When using buffer configuration with timekey, elasticsearch plugin can handle time placeholder:
index_name fluentd.${tag}.%Y%m%d -> index_name fluentd.your.tag.20181011
Signed-off-by: Hiroshi Hatake hatake@clear-code.com