search

fluent / fluentd

Fluentd: Unified Logging Layer (project under CNCF)
https://www.fluentd.org
Apache License 2.0
12.88k stars 1.34k forks source link

Fluentd stops sending logs, but keeps processing #2095

Closed roicostas closed 3 years ago

roicostas commented 6 years ago

Fluentd stops sending logs through out_forward plugin, but it seems to continue proccesing logs because td-agent logs show buffer-overflow errors continuously.

Environment:

td-agent configuration (with names changed), there other 3 files configured with similar configuration which send to different ports:

<source>
    @type tail
    pos_file /var/log/td-agent-stg/xxx.pos
    tag xxxx
    path "/fluentd/loginput/xxxx.txt"
    @label xxx
    format multiline
    format_firstline /^\[[^\]]+\]/
    format1 /^\[(?<logtime>[^\]]+)\] \[(?<thread>.*?)\] \[(?<level>[^\]]*)\] \[MEM=(?<memory>[^\]]*)\] (?<message>.*)$/
</source>
<label xxx>
    <filter **>
        @type record_modifier
        <record>
            utctime ${(DateTime.strptime(record['logtime'], '%Y-%m-%d %H:%M:%S,%N').to_time.utc - Time.now.utc_offset).strftime('%FT%T,%3NZ')}
            yyyy xxxx
        </record>
    </filter>
    <match **>
        @type forward
        send_timeout 60s
        recover_wait 10s
        hard_timeout 60s
        buffer_type memory
        buffer_chunk_limit 8MB
        buffer_queue_limit 28
        flush_interval 1s
        disable_retry_limit true
        max_retry_wait 16
        <server>
            host "xxxx.com"
            port 24225
            weight 60
        </server>
    </match>
</label>
dmnfortytwo commented 6 years ago

Got same problem, but restart did not help, I had to increase buffer -> total_limit_size and after that fluent processed buffers.

2018-08-10 13:45:15 +0000 [info]: starting fluentd-1.2.2 pid=12256 ruby="2.4.4"
2018-08-10 13:45:15 +0000 [info]: spawn command to main:  cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/sbin/td-agent", "--log", "/var/log/td-agent/td-agent.log", "--under-supervis
or"]
2018-08-10 13:45:16 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '2.4.0'
2018-08-10 13:45:16 +0000 [info]: gem 'fluent-plugin-kafka' version '0.7.2'
2018-08-10 13:45:16 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.0.1'
2018-08-10 13:45:16 +0000 [info]: gem 'fluent-plugin-s3' version '1.1.0'
2018-08-10 13:45:16 +0000 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-08-10 13:45:16 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2018-08-10 13:45:16 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.2'
2018-08-10 13:45:16 +0000 [info]: gem 'fluentd' version '1.2.2'
hochdorf commented 6 years ago

I experienced the same problem.

source 'https://rubygems.org'

gem 'fluentd', '<=1.2.2'
gem 'activesupport', '~>5.2.0'
gem 'fluent-plugin-kubernetes_metadata_filter', '~>2.1.2'
gem 'fluent-plugin-elasticsearch', '~>2.10.5'
gem 'fluent-plugin-systemd', '~>1.0.1'
gem 'fluent-plugin-detect-exceptions', '~>0.0.11'
gem 'fluent-plugin-prometheus', '~>1.0.1'
gem 'fluent-plugin-multi-format-parser', '~>1.0.0'
gem 'fluent-plugin-kvp-filter', '~> 0.1.1'
gem 'fluent-plugin-record-modifier', '~>1.0.2'
gem 'fluent-plugin-rewrite-tag-filter', '~>2.0.2'
gem 'fluent-plugin-concat', '~>2.3.0'
gem 'oj', '~>3.6.2'

This is the output part of my config:

<match **>
          @id elasticsearch
          @type elasticsearch
          @log_level info
          include_tag_key true
          host elasticsearch-logging
          port 9200
          time_precision 3 
          logstash_format true
          reconnect_on_error true
          request_timeout 30s     
          <buffer>
            @type file
            path /var/log/fluentd-buffers/kubernetes.system.buffer
            flush_mode interval
            retry_type exponential_backoff
            flush_thread_count 2
            flush_interval 5s
            retry_forever true
            retry_max_interval 30
            chunk_limit_size 10M
            total_limit_size 10G
            queued_chunks_limit_size 100
            overflow_action block
          </buffer>
        </match>

I activated the trace level logging for the output plugin and it seems that fluentd goes into some infinite loop:

2018-09-06 10:52:20 +0000 [trace]: [elasticsearch] enqueueing all chunks in buffer instance=69817523698400
2018-09-06 10:52:20 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:20 +0000 [trace]: [elasticsearch] adding metadata instance=69817523698400 metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag="kubernetes.var.log.containers.service-52efa0cfe96bc1dd783718a0a8fc7d42e46a22e2f6d79aea1b59dc9cddd05a1d.log", variables=nil>
2018-09-06 10:52:20 +0000 [warn]: [elasticsearch] failed to write data into buffer by buffer overflow action=:block
2018-09-06 10:52:20 +0000 [debug]: [elasticsearch] buffer.write is now blocking
2018-09-06 10:52:20 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:21 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:21 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:21 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:21 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:21 +0000 [trace]: [elasticsearch] enqueueing all chunks in buffer instance=69817523698400
2018-09-06 10:52:21 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:21 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:22 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:22 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:22 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:22 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:22 +0000 [trace]: [elasticsearch] enqueueing all chunks in buffer instance=69817523698400
2018-09-06 10:52:22 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:22 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:23 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:23 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:23 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data
2018-09-06 10:52:23 +0000 [trace]: [elasticsearch] sleeping until buffer can store more data

These sleeping, enqueueing all are repeating forever. My elastic cluster is working fine because I have an other fluentd instance, which uses the same config, and it can push the logs to the cluster without any problem. The tried to restart the fluentd but it doesn't help. After the restart it identifies that the buffer is full and stop processing the logs. I tried to increase the total_limit_size and until the buffer became full again fluentd successfully pushed the logs to elastic. For me it seems to be a deadlock which maybe caused by the buffer configuration.

@repeatedly do you have any idea?

hochdorf commented 6 years ago

This is the sigdump:

Sigdump at 2018-09-07 08:52:05 +0000 process 9 (/usr/local/bin/fluentd)
  Thread #<Thread:0x007fd84197e1b0> status=run priority=0
      /var/lib/gems/2.3.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /var/lib/gems/2.3.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /var/lib/gems/2.3.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /var/lib/gems/2.3.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /var/lib/gems/2.3.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /var/lib/gems/2.3.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /var/lib/gems/2.3.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /var/lib/gems/2.3.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /var/lib/gems/2.3.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /var/lib/gems/2.3.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `sleep'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `rescue in write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:853:in `write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:969:in `handle_stream_simple'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:844:in `execute_chunking'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:774:in `emit_buffered'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:159:in `emit_events'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:87:in `emit'
      /var/lib/gems/2.3.0/gems/fluent-plugin-rewrite-tag-filter-2.0.2/lib/fluent/plugin/out_rewrite_tag_filter.rb:74:in `block in process'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event.rb:193:in `block in each'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event.rb:192:in `each'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event.rb:192:in `each'
      /var/lib/gems/2.3.0/gems/fluent-plugin-rewrite-tag-filter-2.0.2/lib/fluent/plugin/out_rewrite_tag_filter.rb:71:in `process'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:763:in `emit_sync'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/out_relabel.rb:29:in `process'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:763:in `emit_sync'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:159:in `emit_events'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:385:in `receive_lines'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:503:in `wrap_receive_lines'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:714:in `block in handle_notify'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:758:in `with_io'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:692:in `handle_notify'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:688:in `block in on_notify'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:688:in `synchronize'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:688:in `on_notify'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:534:in `on_notify'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:507:in `attach'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:264:in `setup_watcher'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:292:in `block in start_watchers'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:278:in `each'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:278:in `start_watchers'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:258:in `refresh_watchers'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_tail.rb:193:in `start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:165:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:153:in `each'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:153:in `block in lifecycle'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:140:in `each'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:140:in `lifecycle'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/root_agent.rb:164:in `start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/engine.rb:274:in `start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/engine.rb:219:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:799:in `run_engine'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:549:in `block in run_worker'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:724:in `main_process'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/supervisor.rb:544:in `run_worker'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
      /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/bin/fluentd:8:in `<top (required)>'
      /usr/local/bin/fluentd:22:in `load'
      /usr/local/bin/fluentd:22:in `<main>'
  Thread #<Thread:0x007fd83f044bc8> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/serverengine-2.0.7/lib/serverengine/socket_manager_unix.rb:77:in `accept'
      /var/lib/gems/2.3.0/gems/serverengine-2.0.7/lib/serverengine/socket_manager_unix.rb:77:in `block in start_server'
  Thread #<Thread:0x007fd83a1e9370> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `sleep'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `rescue in write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:853:in `write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:969:in `handle_stream_simple'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:844:in `execute_chunking'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:774:in `emit_buffered'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:159:in `emit_events'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:87:in `emit'
      /var/lib/gems/2.3.0/gems/fluent-plugin-rewrite-tag-filter-2.0.2/lib/fluent/plugin/out_rewrite_tag_filter.rb:74:in `block in process'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event.rb:107:in `each'
      /var/lib/gems/2.3.0/gems/fluent-plugin-rewrite-tag-filter-2.0.2/lib/fluent/plugin/out_rewrite_tag_filter.rb:71:in `process'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:763:in `emit_sync'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:87:in `emit'
      /var/lib/gems/2.3.0/gems/fluent-plugin-concat-2.3.0/lib/fluent/plugin/filter_concat.rb:297:in `handle_timeout_error'
      /var/lib/gems/2.3.0/gems/fluent-plugin-concat-2.3.0/lib/fluent/plugin/filter_concat.rb:269:in `block (2 levels) in flush_timeout_buffer'
      /var/lib/gems/2.3.0/gems/fluent-plugin-concat-2.3.0/lib/fluent/plugin/filter_concat.rb:262:in `each'
      /var/lib/gems/2.3.0/gems/fluent-plugin-concat-2.3.0/lib/fluent/plugin/filter_concat.rb:262:in `block in flush_timeout_buffer'
      /var/lib/gems/2.3.0/gems/fluent-plugin-concat-2.3.0/lib/fluent/plugin/filter_concat.rb:261:in `synchronize'
      /var/lib/gems/2.3.0/gems/fluent-plugin-concat-2.3.0/lib/fluent/plugin/filter_concat.rb:261:in `flush_timeout_buffer'
      /var/lib/gems/2.3.0/gems/fluent-plugin-concat-2.3.0/lib/fluent/plugin/filter_concat.rb:133:in `on_timer'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd83a1320f8> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `sleep'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `rescue in write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:853:in `write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:969:in `handle_stream_simple'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:844:in `execute_chunking'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:774:in `emit_buffered'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:159:in `emit_events'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.10.5/lib/fluent/plugin/out_elasticsearch.rb:526:in `rescue in send_bulk'
      /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.10.5/lib/fluent/plugin/out_elasticsearch.rb:514:in `send_bulk'
      /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.10.5/lib/fluent/plugin/out_elasticsearch.rb:416:in `write'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1099:in `try_flush'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1378:in `flush_thread_run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:440:in `block (2 levels) in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd83a131d38> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `sleep'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `rescue in write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:853:in `write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:969:in `handle_stream_simple'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:844:in `execute_chunking'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:774:in `emit_buffered'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:159:in `emit_events'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.10.5/lib/fluent/plugin/out_elasticsearch.rb:526:in `rescue in send_bulk'
      /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.10.5/lib/fluent/plugin/out_elasticsearch.rb:514:in `send_bulk'
      /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.10.5/lib/fluent/plugin/out_elasticsearch.rb:416:in `write'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1099:in `try_flush'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1378:in `flush_thread_run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:440:in `block (2 levels) in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd83a131540> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1343:in `sleep'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:1343:in `enqueue_thread_run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd83a131220> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `sleep'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `rescue in write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:853:in `write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:969:in `handle_stream_simple'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:844:in `execute_chunking'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:774:in `emit_buffered'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/out_relabel.rb:29:in `process'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:763:in `emit_sync'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:87:in `emit'
      /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:115:in `emit'
      /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:110:in `block in run'
      /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:135:in `yield_current_entry'
      /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:131:in `watch'
      /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:109:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd83a130e60> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `sleep'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `rescue in write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:853:in `write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:969:in `handle_stream_simple'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:844:in `execute_chunking'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:774:in `emit_buffered'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:159:in `emit_events'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/out_relabel.rb:29:in `process'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:763:in `emit_sync'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:87:in `emit'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_syslog.rb:212:in `emit'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_syslog.rb:204:in `block in message_handler'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/parser_syslog.rb:121:in `parse_plain'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_syslog.rb:188:in `message_handler'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_syslog.rb:150:in `block in start_udp_server'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/server.rb:532:in `on_readable_with_sock'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/io.rb:186:in `on_readable'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd841959400> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `sleep'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:869:in `rescue in write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:853:in `write_guard'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:969:in `handle_stream_simple'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:844:in `execute_chunking'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:774:in `emit_buffered'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/out_relabel.rb:29:in `process'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/output.rb:763:in `emit_sync'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:96:in `emit_stream'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/event_router.rb:87:in `emit'
      /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:115:in `emit'
      /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:110:in `block in run'
      /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:135:in `yield_current_entry'
      /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:131:in `watch'
      /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:109:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd84195d398> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd841972400> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd8419aae18> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd841b70e00> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd841bd4a18> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd841beb830> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd841bf11e0> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd841bf8850> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd840c2ba80> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd837c27c18> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd837c27880> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd837c252b0> status=sleep priority=0
      /usr/lib/ruby/2.3.0/webrick/server.rb:177:in `select'
      /usr/lib/ruby/2.3.0/webrick/server.rb:177:in `block in start'
      /usr/lib/ruby/2.3.0/webrick/server.rb:33:in `start'
      /usr/lib/ruby/2.3.0/webrick/server.rb:164:in `start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd837c240b8> status=sleep priority=0
      /usr/lib/ruby/2.3.0/webrick/server.rb:177:in `select'
      /usr/lib/ruby/2.3.0/webrick/server.rb:177:in `block in start'
      /usr/lib/ruby/2.3.0/webrick/server.rb:33:in `start'
      /usr/lib/ruby/2.3.0/webrick/server.rb:164:in `start'
      /var/lib/gems/2.3.0/gems/fluent-plugin-prometheus-1.0.1/lib/fluent/plugin/in_prometheus.rb:44:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd837c0fc80> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd837c0da48> status=sleep priority=0
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
      /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
      /var/lib/gems/2.3.0/gems/fluentd-1.2.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fd840f1d680> status=sleep priority=0
      /usr/lib/ruby/2.3.0/webrick/httpserver.rb:77:in `wait_readable'
      /usr/lib/ruby/2.3.0/webrick/httpserver.rb:77:in `run'
      /usr/lib/ruby/2.3.0/webrick/server.rb:296:in `block in start_thread'
  Thread #<WEBrick::Utils::TimeoutHandler::Thread:0x007fd840efced0> status=sleep priority=0
      /usr/lib/ruby/2.3.0/webrick/utils.rb:177:in `pop'
      /usr/lib/ruby/2.3.0/webrick/utils.rb:177:in `block in initialize'
  Thread #<Thread:0x007fd83bb24bc8> status=sleep priority=0
      /usr/lib/ruby/2.3.0/webrick/httpserver.rb:77:in `wait_readable'
      /usr/lib/ruby/2.3.0/webrick/httpserver.rb:77:in `run'
      /usr/lib/ruby/2.3.0/webrick/server.rb:296:in `block in start_thread'
  GC stat:
      count: 349
      heap_allocated_pages: 748
      heap_sorted_length: 761
      heap_allocatable_pages: 0
      heap_available_slots: 304898
      heap_live_slots: 304631
      heap_free_slots: 267
      heap_final_slots: 0
      heap_marked_slots: 174652
      heap_swept_slots: 54416
      heap_eden_pages: 748
      heap_tomb_pages: 0
      total_allocated_pages: 756
      total_freed_pages: 8
      total_allocated_objects: 43056779
      total_freed_objects: 42752148
      malloc_increase_bytes: 103680
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 340
      major_gc_count: 9
      remembered_wb_unprotected_objects: 7349
      remembered_wb_unprotected_objects_limit: 14524
      old_objects: 164147
      old_objects_limit: 300994
      oldmalloc_increase_bytes: 120336
      oldmalloc_increase_bytes_limit: 23685480
  Built-in objects:
   304,898: TOTAL
   143,357: T_STRING
    56,890: T_IMEMO
    49,376: T_ARRAY
    19,434: T_HASH
    13,370: T_OBJECT
    11,355: T_DATA
     4,093: T_CLASS
     1,445: T_STRUCT
     1,346: T_NODE
     1,315: T_FILE
     1,009: T_REGEXP
       536: T_ICLASS
       425: T_MATCH
       401: T_MODULE
       308: T_SYMBOL
       161: FREE
        59: T_RATIONAL
         9: T_FLOAT
         8: T_BIGNUM
         1: T_COMPLEX
  All objects:
   110,182: String
    22,185: Array
    13,162: Hash
     5,121: Set
     2,898: Time
     2,352: Class
     1,964: MIME::Type::Columnar
     1,450: Proc
     1,422: Thread::Mutex
     1,421: RubyVM::Env
     1,296: File
     1,284: Fluent::Plugin::Buffer::FileChunk
     1,284: Fluent::Plugin::Buffer::Metadata
     1,027: Fluent::EventTime
     1,000: Regexp
       832: Thread::Backtrace
       694: Errno::ENOENT
       468: Gem::Requirement
       401: Module
       337: Gem::Dependency
       308: Symbol
       197: MatchData
       164: Gem::Version
       139: Fluent::Config::ConfigureProxy
       124: Fluent::Plugin::TailInput::FilePositionEntry
       112: Fluent::Config::Element
       102: File::Stat
       101: EOFError
       101: Encoding
        94: Range
        92: Method
        72: Fluent::Config::Section
        65: Gem::Specification
        65: Gem::StubSpecification::StubLine
        65: Gem::StubSpecification
        59: Rational
        51: Fluent::Plugin::Base::State
        45: FFI::DynamicLibrary::Symbol
        44: FFI::FunctionType
        44: FFI::Function
        33: WEBrick::HTTPVersion
        27: Thread
        24: Strftime
        21: Fluent::PluginHelper::Timer::TimerWatcher
        21: FFI::Type::Builtin
        20: Coolio::Loop
        19: Fluent::TimeParser
        17: Fluent::PluginHelper::EventLoop::DefaultWatcher
        17: RecursiveOpenStruct::DeepDup
        17: OptionParser::Switch::RequiredArgument
        14: Prometheus::Client::LabelSetValidator
        14: Prometheus::Client::Gauge
        13: Logger::Formatter
        13: Strptime
        13: OptionParser::Switch::NoArgument
        12: Fluent::Log
        12: ServerEngine::DaemonLogger
        11: Kubeclient::Common::WatchNotice
        11: Fluent::PluginLogger
        10: WEBrick::HTTPRequest
        10: WEBrick::HTTPResponse
        10: Fluent::Plugin::RegexpParser
        10: Monitor
         9: URI::HTTP
         9: Fluent::MultiEventStream
         9: Fluent::Plugin::TailInput::TailWatcher::RotateHandler
         9: Fluent::Plugin::TailInput::TailWatcher::StatWatcher
         9: Fluent::Plugin::TailInput::TailWatcher
         9: Fluent::EventRouter::Rule
         9: Float
         8: Fluent::Plugin::TailInput
         8: Fluent::Plugin::TailInput::PositionFile
         8: Bignum
         8: Fluent::Registry
         7: IO
         7: Fluent::GlobMatchPattern
         7: Fluent::Plugin::Buffer::BufferOverflowError
         6: Fluent::Plugin::PrometheusInput::MonitorServlet
         6: URI::Generic
         6: Object
         6: Fluent::OneEventStream
         6: OptionParser::OptionMap
         5: Fluent::Plugin::MultilineParser
         5: UnboundMethod
         5: Fluent::EventRouter::Pipeline::FilterOptimizer
         5: Fluent::EventRouter::Pipeline
         5: HTTP::Headers
         5: Fluent::Plugin::TailInput::TailWatcher::LineBufferTimerFlusher
         4: Coolio::IO::Watcher
         4: Fluent::Plugin::MonitorAgentInput
         4: FFI::DynamicLibrary
         4: TCPSocket
         4: LruRedux::TTL::ThreadSafeCache
         4: Concurrent::Map
         3: ThreadGroup
         3: Fluent::PluginHelper::Server::ServerInfo
         3: UDPSocket
         3: TCPServer
         3: Faraday::Utils::Headers
         3: FFI::StructLayout::Array
         3: FFI::ArrayType
         3: FFI::Pointer
         3: URI::HTTPS
         3: Fiber
         3: Fluent::PluginHelper::RecordAccessor::Accessor
         3: OptionParser::List
         3: IRB::Notifier::LeveledNotifier
         3: JSON::Ext::Generator::State
         2: IO::Buffer
         2: Fluent::PluginHelper::Server::EventHandler::UDPServer
         2: WEBrick::HTTPServer::MountTable
         2: Thread::SizedQueue
         2: WEBrick::HTTPServer
         2: WEBrick::Log
         2: Fluent::Plugin::SyslogParser
         2: BigDecimal
         2: Fluent::Plugin::TailInput::TailWatcher::FIFO
         2: Fluent::Plugin::TailInput::TailWatcher::IOHandler
         2: FFI::Type::Mapped
         2: Fluent::Plugin::SystemdInput
         2: Fluent::Plugin::RecordModifierFilter::DynamicExpander
         2: Fluent::Plugin::SystemdEntryMutator::Options
         2: Fluent::Plugin::SystemdEntryMutator
         2: Fluent::PluginHelper::Storage::StorageState
         2: Fluent::PluginHelper::Storage::PersistentWrapper
         2: HTTP::Response::Status
         2: HTTP::URI
         2: HTTP::Response
         2: HTTP::Response::Body
         2: Fluent::AllMatchPattern
         2: OpenSSL::SSL::SSLSocket
         2: OpenSSL::SSL::SSLContext
         2: HTTP::Timeout::Null
         2: HTTP::Parser
         2: HTTP::Response::Parser
         2: HTTP::Connection
         2: HTTP::Options
         2: HTTP::Client
         2: Kubeclient::Common::WatchStream
         2: Fluent::Plugin::Output::FlushThreadState
         2: Fluent::Plugin::LocalStorage
         2: Fluent::Plugin::ElasticsearchOutput::RetryStreamError
         2: Fluent::Plugin::ElasticsearchErrorHandler
         2: Systemd::JournalEntry
         2: Systemd::Id128::Native::Id128
         2: FFI::MemoryPointer
         2: Systemd::Journal
         2: WEBrick::Utils::TimeoutHandler::Thread
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: ThreadSafe::Cache
         2: MessagePack::Factory
         2: Fluent::EventRouter::MatchCache
         2: Fluent::EventRouter
         2: Fluent::NoMatchMatch
         2: MIME::Types::Container
         2: OpenSSL::PKey::DH
         1: Enumerator
         1: Coolio::TCPServer
         1: Faraday::Adapter::Excon
         1: Elasticsearch::Transport::Client
         1: Elasticsearch::Transport::Transport::Sniffer
         1: Elasticsearch::Transport::Transport::Serializer::MultiJson
         1: Elasticsearch::Transport::Transport::Connections::Selector::RoundRobin
         1: Elasticsearch::Transport::Transport::Connections::Collection
         1: Elasticsearch::Transport::Transport::Connections::Connection
         1: Faraday::RackBuilder::Handler
         1: Faraday::RackBuilder
         1: Faraday::Utils::ParamsHash
         1: Faraday::SSLOptions
         1: Faraday::RequestOptions
         1: Faraday::Connection
         1: Elasticsearch::Transport::Transport::HTTP::Faraday
         1: Fluent::Plugin::FileBuffer
         1: Fluent::Plugin::PrometheusTailMonitorInput
         1: Fluent::Plugin::PrometheusOutputMonitorInput
         1: Fluent::Plugin::PrometheusMonitorInput
         1: Fluent::Plugin::ElasticsearchOutput
         1: Prometheus::Client::Registry
         1: Fluent::Plugin::PrometheusInput
         1: FFI::VariadicInvoker
         1: FFI::Enum
         1: FFI::Enums
         1: FFI::StructByValue
         1: FFI::StructLayout
         1: Fluent::Plugin::KVPFilter
         1: Fluent::Plugin::RecordModifierFilter
         1: Fluent::Plugin::NullOutput
         1: MultiJson::Adapters::Oj
         1: Fluent::PluginHelper::Server::UDPCallbackSocket
         1: UNIXSocket
         1: Psych::Handler::DumperOptions
         1: ArgumentError
         1: Fluent::Plugin::SyslogInput
         1: Kubeclient::Client
         1: KubernetesMetadata::Stats
         1: RestClient::Resource
         1: I18n::Config
         1: ActiveSupport::Inflector::Inflections::Uncountables
         1: ActiveSupport::Inflector::Inflections
         1: Fluent::Plugin::RelabelOutput
         1: ActiveSupport::Deprecation
         1: Fluent::Plugin::ConcatFilter
         1: ActiveSupport::Notifications::Fanout
         1: Fluent::Plugin::ForwardInput
         1: Fluent::Plugin::JSONParser
         1: Fluent::Plugin::RewriteTagFilterOutput
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: Process::Status
         1: OptionParser
         1: Fluent::Plugin::MultiFormatParser
         1: Fluent::EngineClass
         1: Fluent::Compat::NullOutputChain
         1: Fluent::Plugin::KubernetesMetadataFilter
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Fluent::RootAgent
         1: UNIXServer
         1: ServerEngine::SocketManager::Server
         1: Fluent::SystemConfig
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: IRB::Notifier::NoMsgNotifier
         1: Fluent::Label
         1: StringScanner
         1: MIME::Types
         1: MIME::Types::WarnLogger::WarnLogDevice
         1: MIME::Types::WarnLogger
         1: OptionParser::CompletingHash
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: #<Class:0x007fd840d5a618>
         1: Gem::Platform
         1: Thread::Queue
         1: WEBrick::Utils::TimeoutHandler
         1: OpenSSL::X509::Store
         1: Fluent::Plugin::ParserFilter
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
         1: fatal
         1: Data
         1: Gem::PathSupport
         1: #<Class:0x007fd841b5ab50>
  String 3,223,527 bytes
   Array 0 elements
    Hash 1 pairs
amitdhawan commented 5 years ago

@hochdorf is this resolved?

hochdorf commented 5 years ago

@amitdhawan not really. We changed our configuration to use throw_exception as overflow_action instead of the block value. So now we can have data loss, which is not so optimal, but at least fluentd doesn't stop working. Since it is not so easy to reproduce the situation, I haven't tried the latest fluentd version with the previous config.

liuchintao commented 5 years ago

I met the similar problem, fluentd logs that SLUB: Unable to allocate memory on node -1 (gfp=0x2080020) cache: nf_conntrack_213(39301:87632d18f091508a406615a851ad686609d42a13e6aeadc59b66a53c9d6574cf), object size: 312, buffer size: 320, default order: 1, min order: 0 node 0: slabs: 20, objs: 500, free: 0
After logging a series of [elasticsearch] failed to write data into buffer by buffer overflow action=:block fluentd do not work any more, but the process is healthy :-(

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

github-actions[bot] commented 3 years ago

This issue was automatically closed because of stale in 30 days