Closed bhaveshmodiatekaplus closed 3 years ago
It seems S3 plugin can't send data to S3. No other error logs?
please find buffer configuration below
<buffer tag,time>
@type file
path C:\Users\bhavesh.modi\Desktop\Log\Fluentd\s3
flush_at_shutdown true
flush_mode interval
flush_interval 2m
chunk_limit_size 10MB
total_limit_size 1024MB
</buffer>
<format>
@type single_value
</format>
time_slice_format %Y/%m/%d/stats_log
</match>
please find below console log
2019-11-22 00:46:18 +0530 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2019-11-22 00:46:18 +0530 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="can't create buffer metadata for C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.*.log. Stop creating buffer files: error = Too many open files @ rb_sysopen - C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.b597e021be1de0cf04debc549c379ceae.log.meta" location="C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:304:in `rescue in create_new_chunk'" tag="platform"
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:304:in `rescue in create_new_chunk'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:293:in `create_new_chunk'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:54:in `initialize'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buf_file.rb:173:in `new'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buf_file.rb:173:in `generate_chunk'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer.rb:555:in `block in write_once'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer.rb:555:in `write_once'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer.rb:278:in `block in write'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer.rb:276:in `each'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer.rb:276:in `write'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:951:in `block in handle_stream_with_custom_format'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:877:in `write_guard'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:950:in `handle_stream_with_custom_format'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:869:in `execute_chunking'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:797:in `emit_buffered'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/event_router.rb:97:in `emit_stream'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:410:in `receive_lines'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:528:in `wrap_receive_lines'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:762:in `block in handle_notify'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:806:in `with_io'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:742:in `handle_notify'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:738:in `block in on_notify'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:738:in `synchronize'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:738:in `on_notify'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:558:in `on_notify'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:532:in `attach'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:281:in `setup_watcher'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:312:in `block in start_watchers'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:298:in `each'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:298:in `start_watchers'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:275:in `refresh_watchers'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.4/lib/cool.io/loop.rb:88:in `run_once'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.4/lib/cool.io/loop.rb:88:in `run'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
2019-11-22 00:46:18 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-11-22 00:46:19 +0530 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2019-11-22 00:46:19 +0530 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="can't create buffer metadata for C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.*.log. Stop creating buffer files: error = Too many open files @ rb_sysopen - C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.b597e021d1200954f04b678072591e26b.log.meta" location="C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:304:in `rescue in create_new_chunk'" tag="platform"
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:304:in `rescue in create_new_chunk'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:293:in `create_new_chunk'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:54:in `initialize'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buf_file.rb:173:in `new'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buf_file.rb:173:in `generate_chunk'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer.rb:555:in `block in write_once'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer.rb:555:in `write_once'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer.rb:278:in `block in write'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer.rb:276:in `each'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer.rb:276:in `write'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:951:in `block in handle_stream_with_custom_format'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:877:in `write_guard'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:950:in `handle_stream_with_custom_format'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:869:in `execute_chunking'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/output.rb:797:in `emit_buffered'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/event_router.rb:97:in `emit_stream'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:410:in `receive_lines'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:528:in `wrap_receive_lines'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:762:in `block in handle_notify'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:806:in `with_io'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:742:in `handle_notify'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:738:in `block in on_notify'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:738:in `synchronize'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:738:in `on_notify'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:558:in `on_notify'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/in_tail.rb:644:in `on_timer'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.4/lib/cool.io/loop.rb:88:in `run_once'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.4/lib/cool.io/loop.rb:88:in `run'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
2019-11-22 00:46:19 +0530 [warn]: #0 C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2019-11-22 00:46:20 +0530 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2019-11-22 00:46:20 +0530 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="can't create buffer metadata for C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.*.log. Stop creating buffer files: error = Too many open files @ rb_sysopen - C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.b597e021e069b384239afeda05dc11763.log.meta" location="C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:304:in `rescue in create_new_chunk'" tag="platform"
2019-11-22 00:46:20 +0530 [warn]: #0 suppressed same stacktrace
2019-11-22 00:46:21 +0530 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2019-11-22 00:46:21 +0530 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="can't create buffer metadata for C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.*.log. Stop creating buffer files: error = Too many open files @ rb_sysopen - C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.b597e021ef994cd0eef396c32f0ee87be.log.meta" location="C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:304:in `rescue in create_new_chunk'" tag="platform"
2019-11-22 00:46:21 +0530 [warn]: #0 suppressed same stacktrace
2019-11-22 00:46:22 +0530 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2019-11-22 00:46:22 +0530 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="can't create buffer metadata for C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.*.log. Stop creating buffer files: error = Too many open files @ rb_sysopen - C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.b597e021ff0e9a78a899b6aa48487dc5f.log.meta" location="C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:304:in `rescue in create_new_chunk'" tag="platform"
2019-11-22 00:46:22 +0530 [warn]: #0 suppressed same stacktrace
2019-11-22 00:46:23 +0530 [warn]: #0 failed to write data into buffer by buffer overflow action=:throw_exception
2019-11-22 00:46:23 +0530 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="can't create buffer metadata for C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.*.log. Stop creating buffer files: error = Too many open files @ rb_sysopen - C:\\Users\\name\\Desktop\\Log\\Fluentd\\s3/buffer.b597e0220e5603f7a0ec7e07bd6078f5f.log.meta" location="C:/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.0/lib/fluent/plugin/buffer/file_chunk.rb:304:in `rescue in create_new_chunk'" tag="platform"
2019-11-22 00:46:23 +0530 [warn]: #0 suppressed same stacktrace
@bhaveshmodiatekaplus Were you able to solve this? My chunk size is even smaller 256KB.
This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days
This issue was automatically closed because of stale in 30 days
Hi, I'm trying to export log Application logs to S3 every 5min, but i wanted to store logs in date wise folder like below ex : S3 path - platformLog/web/ 2019/11/20/platform_log_0.gz 2019/11/20/platform_log_1.gz 2019/11/20/platform_log_2.gz 2019/11/21/platform_log_0.gz 2019/11/21/platform_log_1.gz.
When i use 'time' argument in buffer section , it is creating lot of meta files and throwing buffer error on console, Can any one tell whats correct parameter which we can use here for above folder structure. Thanks in advance.
Error on console