search

awslabs / aws-fluent-plugin-kinesis

Amazon Kinesis output plugin for Fluentd
Apache License 2.0
293 stars 96 forks source link

Compression in "buffer" section doesn't work #151

Closed tonyjiang closed 5 years ago

tonyjiang commented 6 years ago

I have the following configuration inside the match section. I can see the compressed file in the specified path, but nothing gets sent to Kinesis (in contrast, data get sent to Kinesis without this buffer section). If this is expected behavior, it should be clearly documented in README. If it's not, it should get fixed.

<match logs.to.kinesis>
  @type kinesis_streams
  ...
  <buffer>
    @type file
    path /var/log/td-agent/buffer
    compress gzip
    chunk_limit_size 1M
    total_limit_size 64G
  </buffer>
  ...
</match>

Here are the error messages in td-agent.log:

2018-06-27 00:24:26 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2018-06-27 00:24:27 +0000 chunk="56f94a2ff6fbc7a508d7af4c05542021" error_class=Zlib::GzipFile::Error error="not in gzip format"
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/compressable.rb:77:in `initialize'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/compressable.rb:77:in `new'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/compressable.rb:77:in `block in io_decompress'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/compressable.rb:76:in `loop'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/compressable.rb:76:in `io_decompress'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/compressable.rb:38:in `decompress'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/buffer/chunk.rb:193:in `block in open'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/buffer/file_chunk.rb:172:in `open'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/buffer/chunk.rb:187:in `open'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/event.rb:322:in `each'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-2.1.1/lib/fluent/plugin/kinesis_helper/api.rb:75:in `each'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-2.1.1/lib/fluent/plugin/kinesis_helper/api.rb:75:in `split_to_batches'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-2.1.1/lib/fluent/plugin/kinesis.rb:147:in `write_records_batch'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-kinesis-2.1.1/lib/fluent/plugin/out_kinesis_streams.rb:43:in `write'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/compat/output.rb:131:in `write'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1099:in `try_flush'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1378:in `flush_thread_run'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:440:in `block (2 levels) in start'
  2018-06-27 00:24:26 +0000 [warn]: #0 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2018-06-27 00:24:27 +0000 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2018-06-27 00:24:27 +0000 chunk="56f94a2ff6fbc7a508d7af4c05542021" error_class=Zlib::GzipFile::Error error="not in gzip format"

Seems Kinesis plugin is not able to handle gzip data correctly.

But the files in the specified path (/var/log/td-agent/buffer) indeed don't have the usual .gz extension. I can't uncompress them with gzip command. They somehow come in pairs like this:

-rw-r--r-- 1 td-agent td-agent 301552 Jun 27 00:24 buffer.q56f94a2ff6fbc7a508d7af4c05542021.log
-rw-r--r-- 1 td-agent td-agent     66 Jun 27 00:24 buffer.q56f94a2ff6fbc7a508d7af4c05542021.log.meta
-rw-r--r-- 1 td-agent td-agent 419055 Jun 27 00:25 buffer.q56f94a6a1d3cc39a52b39268a14cbf4d.log
-rw-r--r-- 1 td-agent td-agent     66 Jun 27 00:25 buffer.q56f94a6a1d3cc39a52b39268a14cbf4d.log.meta
-rw-r--r-- 1 td-agent td-agent 433263 Jun 27 00:26 buffer.q56f94aa4529da96ae55d8cf06f1de3f8.log
-rw-r--r-- 1 td-agent td-agent     66 Jun 27 00:26 buffer.q56f94aa4529da96ae55d8cf06f1de3f8.log.meta
-rw-r--r-- 1 td-agent td-agent 421523 Jun 27 00:27 buffer.q56f94ade769fa80076689762cb5304a7.log
-rw-r--r-- 1 td-agent td-agent     66 Jun 27 00:27 buffer.q56f94ade769fa80076689762cb5304a7.log.meta
-rw-r--r-- 1 td-agent td-agent 431826 Jun 27 00:28 buffer.q56f94b18a64f3c1e5b8d41d23e0bb4f0.log
-rw-r--r-- 1 td-agent td-agent     66 Jun 27 00:28 buffer.q56f94b18a64f3c1e5b8d41d23e0bb4f0.log.meta
-rw-r--r-- 1 td-agent td-agent  90776 Jun 27 00:29 buffer.q56f94b52e34863822fa2d1eb8dbcc23a.log
-rw-r--r-- 1 td-agent td-agent     65 Jun 27 00:29 buffer.q56f94b52e34863822fa2d1eb8dbcc23a.log.meta
-rw-r--r-- 1 td-agent td-agent  23088 Jun 27 00:42 buffer.q56f94e43414d1128b1cfa901c22cdb4e.log
-rw-r--r-- 1 td-agent td-agent     64 Jun 27 00:42 buffer.q56f94e43414d1128b1cfa901c22cdb4e.log.meta
-rw-r--r-- 1 td-agent td-agent   8265 Jun 27 00:43 buffer.q56f94e7d770c5204e4cbffa3c4f7f2d8.log
-rw-r--r-- 1 td-agent td-agent     64 Jun 27 00:43 buffer.q56f94e7d770c5204e4cbffa3c4f7f2d8.log.meta
-rw-r--r-- 1 td-agent td-agent    381 Jun 27 00:47 buffer.q56f94f662face0adfa2ecfd882b39f6a.log
-rw-r--r-- 1 td-agent td-agent     64 Jun 27 00:47 buffer.q56f94f662face0adfa2ecfd882b39f6a.log.meta

Any insight is much appreciated! Tony

riywo commented 6 years ago

Could you give us the versions of both fluentd and plugin?

simukappu commented 5 years ago

Closing this issue for now. Please reopen if required.