search

fluent / fluentd

Fluentd: Unified Logging Layer (project under CNCF)
https://www.fluentd.org
Apache License 2.0
12.84k stars 1.34k forks source link

Fluentd 0.14.11 deadlocked in file output #1549

Closed hfinucane closed 6 years ago

hfinucane commented 7 years ago

td-agent 3/fluentd 0.14.11, running on CentOS 6, writing to a file. Senders were all buffering. Multiple sigdumps were identical other than the allocation count, so I feel fairly confident things were deadlocked.

I had a hard time killing the daemon when it was in this state

2017-04-18 21:44:20 +0000 [warn]: thread doesn't exit correctly (killed or other reason) plugin=Fluent::Plugin::FileOutput title=:flush_thread_0 thread=#<Thread:0x007f092a15c720@/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:58 aborting> error=nil

We rolled back to td-agent 2/fluentd 0.12, and may have seen the same thing, but I am not sure, and we can no longer reproduce the problem.

Sigdump at 2017-04-18 20:47:33 +0000 process 3303 (/usr/sbin/td-agent)
  Thread #<Thread:0x007f0931faf988> status=run priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/engine.rb:179:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/engine.rb:179:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/supervisor.rb:725:in `run_engine'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/supervisor.rb:478:in `block in run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/supervisor.rb:650:in `main_process'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/supervisor.rb:473:in `run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/command/fluentd.rb:288:in `<top (required)>'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/bin/fluentd:5:in `<top (required)>'
      /opt/td-agent/embedded/bin/fluentd:22:in `load'
      /opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
      /usr/sbin/td-agent:7:in `load'
      /usr/sbin/td-agent:7:in `<main>'
  Thread #<Thread:0x007f092b55d940> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1250:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1250:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:398:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
  Thread #<Thread:0x007f092b55d508> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1207:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1207:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
  Thread #<Thread:0x007f092a15c720> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:900:in `commit_write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1013:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1234:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:398:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
  Thread #<Thread:0x007f092a15c450> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:353:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1197:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
  Thread #<Thread:0x007f092741f898> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1250:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1250:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:398:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
  Thread #<Thread:0x007f092741f398> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1207:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:1207:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
  Thread #<Thread:0x007f092741eec0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/event_loop.rb:77:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
  Thread #<Thread:0x007f0927413ef8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_monitor_agent.rb:234:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
  Thread #<Thread:0x007f0927413638> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/event_loop.rb:77:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
  Thread #<Thread:0x007f0927df3ec8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:457:in `write_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:207:in `block in write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:206:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/buffer.rb:206:in `write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:848:in `block in handle_stream_with_custom_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:783:in `write_guard'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:847:in `handle_stream_with_custom_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:775:in `execute_chunking'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/output.rb:710:in `emit_buffered'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/event_router.rb:90:in `emit_stream'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:292:in `on_message'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:203:in `block in handle_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:240:in `block (3 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:239:in `feed_each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:239:in `block (2 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin/in_forward.rb:248:in `block in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/server.rb:465:in `on_read_without_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/io.rb:123:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/io.rb:186:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.4.5/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/event_loop.rb:77:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/plugin_helper/thread.rb:66:in `block in thread_create'
  Thread #<Thread:0x007f0927df16c8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/engine.rb:149:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.11/lib/fluent/engine.rb:149:in `log_event_loop'
  GC stat:
      count: 3840
      heap_allocated_pages: 273
      heap_sorted_length: 273
      heap_allocatable_pages: 0
      heap_available_slots: 111273
      heap_live_slots: 110894
      heap_free_slots: 379
      heap_final_slots: 0
      heap_marked_slots: 66052
      heap_eden_pages: 273
      heap_tomb_pages: 0
      total_allocated_pages: 273
      total_freed_pages: 0
      total_allocated_objects: 142062923
      total_freed_objects: 141952029
      malloc_increase_bytes: 35536
      malloc_increase_bytes_limit: 29739514
      minor_gc_count: 3648
      major_gc_count: 192
      remembered_wb_unprotected_objects: 1658
      remembered_wb_unprotected_objects_limit: 2376
      old_objects: 63003
      old_objects_limit: 121542
      oldmalloc_increase_bytes: 35984
      oldmalloc_increase_bytes_limit: 92131169
  Built-in objects:
   111,273: TOTAL
    27,789: T_STRING
    22,271: T_DATA
    20,514: T_ARRAY
    18,339: T_IMEMO
     6,865: T_RATIONAL
     6,812: T_BIGNUM
     2,492: T_CLASS
     1,848: T_OBJECT
     1,241: T_NODE
     1,114: T_HASH
       590: T_REGEXP
       277: FREE
       277: T_ICLASS
       271: T_FILE
       258: T_STRUCT
       227: T_MODULE
        73: T_SYMBOL
         9: T_FLOAT
         5: T_MATCH
         1: T_COMPLEX
  All objects:
    28,024: String
    11,516: Time
     9,453: Array
     3,711: Rational
     3,658: Integer
     1,431: Class
       993: Hash
       665: Proc
       600: Gem::Requirement
       590: Regexp
       430: Gem::Dependency
       227: Module
       217: File
       208: Thread::Mutex
       180: Fluent::Plugin::Buffer::Metadata
       177: Gem::Version
       106: Fluent::Plugin::Buffer::FileChunk
       101: Encoding
        92: Coolio::IO::Watcher
        77: Gem::Specification
        77: Gem::StubSpecification::StubLine
        77: Gem::StubSpecification
        75: Fluent::Config::ConfigureProxy
        73: Symbol
        56: Range
        46: IO::Buffer
        45: Fluent::PluginHelper::Server::TCPCallbackSocket
        45: Fluent::PluginHelper::Server::EventHandler::TCPServer
        45: TCPSocket
        42: MessagePack::Buffer
        42: MessagePack::Unpacker
        27: MatchData
        24: Fluent::Config::Section
        17: Fluent::Config::Element
        16: OptionParser::Switch::RequiredArgument
        13: Thread::Backtrace
        13: Fluent::Plugin::Base::State
        12: Thread
        12: OptionParser::Switch::NoArgument
        11: Method
         9: Float
         8: Fluent::Registry
         6: OptionParser::OptionMap
         5: IO
         4: Monitor
         4: Object
         3: TCPServer
         3: Fluent::PluginHelper::EventLoop::DefaultWatcher
         3: Coolio::Loop
         3: Fluent::Plugin::Output::FlushThreadState
         3: Fluent::EventRouter::Rule
         3: Fluent::TimeFormatter
         3: Fluent::Plugin::OutFileFormatter
         3: Fluent::Plugin::FileBuffer
         3: Fluent::Plugin::FileOutput
         3: Fluent::PluginHelper::Server::ServerInfo
         3: OptionParser::List
         3: IRB::Notifier::LeveledNotifier
         3: JSON::Ext::Generator::State
         2: Zlib::GzipWriter
         2: Coolio::TCPServer
         2: ThreadGroup
         2: Fluent::GlobMatchPattern
         2: MessagePack::Factory
         2: OpenSSL::PKey::DH
         2: ThreadSafe::Cache
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: UnboundMethod
         1: WEBrick::HTTPServer::MountTable
         1: Fluent::Plugin::SyslogInput
         1: Thread::SizedQueue
         1: WEBrick::HTTPServer
         1: WEBrick::Log
         1: Fluent::Plugin::ForwardInput
         1: Fluent::MessagePackEventStream
         1: EOFError
         1: Strptime
         1: Fluent::TimeParser
         1: Fluent::Plugin::SyslogParser
         1: Fluent::AllMatchPattern
         1: Fluent::EventRouter::MatchCache
         1: Fluent::EventRouter
         1: Fluent::Agent::NoMatchMatch
         1: Fluent::RootAgent
         1: Fluent::SystemConfig
         1: Fluent::EngineClass
         1: OpenSSL::X509::Store
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Fluent::Compat::NullOutputChain
         1: Fluent::PluginHelper::Server::EventHandler::UDPServer
         1: UDPSocket
         1: WEBrick::HTTPVersion
         1: Set
         1: OptionParser
         1: Fluent::Plugin::MonitorAgentInput
         1: IRB::Notifier::NoMsgNotifier
         1: #<Class:0x007f092b4a7d98>
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: Gem::PathSupport
         1: #<Class:0x007f092b51e560>
         1: Gem::Platform
         1: fatal
         1: Data
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: Fluent::Log
         1: Logger::Formatter
         1: ServerEngine::DaemonLogger
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: OptionParser::CompletingHash
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
  String 19,199,333 bytes
   Array 1 elements
    Hash 14 pairs
repeatedly commented 7 years ago

Thanks for the report. I first heard this out_file issue. Is this reproducible? And does v0.14.15 also have same problem?

hfinucane commented 7 years ago

It was reproduceable in my environment for about a day, but I can no longer reproduce after rolling back and then forward again. I was hoping the sigdump would have enough information to debug.

hfinucane commented 7 years ago

I caught something similar running td-agent 0.14.16-

Sigdump at 2017-06-07 20:02:50 +0000 process 1160 (/usr/sbin/td-agent)
  Thread #<Thread:0x007f54d4db60c0> status=run priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/engine.rb:224:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/engine.rb:224:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/supervisor.rb:737:in `run_engine'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/supervisor.rb:503:in `block in run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/supervisor.rb:661:in `main_process'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/supervisor.rb:498:in `run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/bin/fluentd:5:in `<top (required)>'
      /opt/td-agent/embedded/bin/fluentd:22:in `load'
      /opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
      /usr/sbin/td-agent:7:in `load'
      /usr/sbin/td-agent:7:in `<main>'
  Thread #<Thread:0x007f54cb4a33d8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cb4a2fc8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1241:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1241:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4f4ae8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4efdb8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4ef688> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4ee210> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4e6ba0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4e5278> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4de018> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4dce98> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cd1be4e0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4d7808> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4d5be8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4d4518> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4cfdd8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cc4cf770> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca2675c0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca267250> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca266f08> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca266b70> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca2666e8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca266260> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca265dd8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca2659c8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca265338> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca264aa0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca264780> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca264398> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca267b60> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca257968> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca2575a8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca257210> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca256e50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca256a40> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca256658> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:353:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca255898> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca255528> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca255258> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca255028> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca254dd0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca254b50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca2548d0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca254678> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca2543a8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ca2540d8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d5bf58> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d5b5a8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d5ad88> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d5a400> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d59a50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d592a8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d58678> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d6fbc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d6e090> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d6ca88> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d6c2e0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d7e238> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d7d450> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d7c7f8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d7c050> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d8e660> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d9e268> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d9c0a8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4dad088> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4dbfa30> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4dcfbd8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4dccd70> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1284:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4dcc3c0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1241:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:1241:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4dde160> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cebafe30> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54cebae170> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54ceacdad0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:457:in `write_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:207:in `block in write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:206:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/buffer.rb:206:in `write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:879:in `block in handle_stream_with_custom_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:805:in `write_guard'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:878:in `handle_stream_with_custom_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:797:in `execute_chunking'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/output.rb:732:in `emit_buffered'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/event_router.rb:159:in `emit_events'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/event_router.rb:96:in `emit_stream'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:300:in `on_message'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007f54d4d641a8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/engine.rb:192:in `sleep'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.16/lib/fluent/engine.rb:192:in `log_event_loop'
  GC stat:
      count: 511419
      heap_allocated_pages: 721
      heap_sorted_length: 809
      heap_allocatable_pages: 0
      heap_available_slots: 293869
      heap_live_slots: 292972
      heap_free_slots: 897
      heap_final_slots: 0
      heap_marked_slots: 197865
      heap_eden_pages: 721
      heap_tomb_pages: 0
      total_allocated_pages: 1793
      total_freed_pages: 1072
      total_allocated_objects: 52329380397
      total_freed_objects: 52329087425
      malloc_increase_bytes: 139456
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 500707
      major_gc_count: 10712
      remembered_wb_unprotected_objects: 786
      remembered_wb_unprotected_objects_limit: 1572
      old_objects: 195575
      old_objects_limit: 341804
      oldmalloc_increase_bytes: 414784
      oldmalloc_increase_bytes_limit: 19088575
  Built-in objects:
   293,869: TOTAL
   131,344: T_STRING
    44,559: T_DATA
    36,110: T_ARRAY
    25,132: T_HASH
    17,995: T_IMEMO
    14,368: T_RATIONAL
    14,315: T_BIGNUM
     3,619: T_OBJECT
     2,421: T_CLASS
     1,392: T_NODE
       821: FREE
       578: T_REGEXP
       554: T_STRUCT
       283: T_ICLASS
       230: T_MODULE
        96: T_SYMBOL
        38: T_FILE
         9: T_FLOAT
         4: T_MATCH
         1: T_COMPLEX
  All objects:
   130,658: String
    24,558: Hash
    21,772: Time
    20,407: Array
     7,214: Rational
     7,160: Integer
     1,382: Class
     1,024: Fluent::EventRouter::Pipeline::FilterOptimizer
     1,024: Fluent::EventRouter::Pipeline
       590: Proc
       578: Regexp
       533: Gem::Requirement
       420: Fluent::Plugin::Buffer::Metadata
       371: Gem::Dependency
       230: Module
       178: Gem::Version
       131: Thread::Mutex
       101: Encoding
        96: Symbol
        80: Thread::Backtrace
        74: Thread
        73: Gem::Specification
        73: Gem::StubSpecification::StubLine
        73: Gem::StubSpecification
        72: Fluent::Config::ConfigureProxy
        65: Fluent::Plugin::Output::FlushThreadState
        48: Range
        28: Fluent::Config::Section
        26: MatchData
        22: Coolio::IO::Watcher
        21: Fluent::Config::Element
        17: Method
        17: OptionParser::Switch::RequiredArgument
        15: File
        15: Fluent::Plugin::Base::State
        13: OptionParser::Switch::NoArgument
        11: IO::Buffer
        10: Fluent::PluginHelper::Server::TCPCallbackSocket
        10: Fluent::PluginHelper::Server::EventHandler::TCPServer
        10: TCPSocket
         9: Float
         8: Fluent::Registry
         6: OptionParser::OptionMap
         5: Fluent::Plugin::Buffer::FileChunk
         5: EOFError
         5: Fluent::EventRouter::Rule
         5: IO
         4: Monitor
         4: Fluent::GlobMatchPattern
         4: Object
         3: Coolio::Loop
         3: Fluent::Plugin::FileOutput
         3: Fluent::Plugin::FileBuffer
         3: OptionParser::List
         3: Fluent::TimeFormatter
         3: Fluent::Plugin::OutFileFormatter
         3: IRB::Notifier::LeveledNotifier
         3: JSON::Ext::Generator::State
         3: Fluent::PluginHelper::EventLoop::DefaultWatcher
         3: Fluent::PluginHelper::Server::ServerInfo
         3: TCPServer
         2: Fluent::EventTime
         2: MessagePack::Buffer
         2: MessagePack::Unpacker
         2: Fluent::Plugin::IllumioUnwrapFilter
         2: OptionParser::Switch::OptionalArgument
         2: ThreadSafe::Cache
         2: OptionParser::Switch::PlacedArgument
         2: MessagePack::Factory
         2: OpenSSL::PKey::DH
         2: UnboundMethod
         2: Coolio::TCPServer
         2: ThreadGroup
         2: Fluent::PluginHelper::Server::UDPCallbackSocket
         1: Fluent::MultiEventStream
         1: Fluent::MessagePackEventStream
         1: Fluent::Plugin::SyslogInput
         1: Fluent::Compat::NullOutputChain
         1: Strptime
         1: Fluent::TimeParser
         1: Fluent::Plugin::SyslogParser
         1: ArgumentError
         1: Fluent::Plugin::MonitorAgentInput
         1: OpenSSL::X509::Store
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: OptionParser
         1: WEBrick::HTTPVersion
         1: Fluent::EngineClass
         1: Set
         1: Fluent::Log
         1: Logger::Formatter
         1: ServerEngine::DaemonLogger
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: Fluent::AllMatchPattern
         1: Fluent::Plugin::ForwardInput
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: IRB::Notifier::NoMsgNotifier
         1: Fluent::EventRouter::MatchCache
         1: Fluent::EventRouter
         1: Fluent::NoMatchMatch
         1: Fluent::RootAgent
         1: Fluent::SystemConfig
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: Gem::PathSupport
         1: Gem::Platform
         1: #<Class:0x007f54ce2d04b8>
         1: #<Class:0x007f54ce30d778>
         1: fatal
         1: Data
         1: WEBrick::HTTPServer::MountTable
         1: OptionParser::CompletingHash
         1: Fluent::PluginHelper::Server::EventHandler::UDPServer
         1: UDPSocket
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
         1: WEBrick::Log
         1: Thread::SizedQueue
         1: WEBrick::HTTPServer
  String 4,243,574 bytes
   Array 4 elements
    Hash 1 pairs

There are a ton more threads, and I'm not sure if it's the same or not, but the symptoms are the same- deadlocked daemon with no files getting written.

dasterintempus commented 7 years ago

I am also experiencing what appears to be the same issue. RHEL 6, fluentd gem 0.14.16. sigdump from the deadlocked worker is below, followed by the sigdump from a functional worker (note the lack of locks).

Edit: Moved to attachments instead of giant code blocks.

orig_deadlock.txt functioning_threaddump.txt

dasterintempus commented 7 years ago

Here is another threaddump, from after all 4 worker processes deadlocked. threaddump.txt

I have just updated to fluentd 0.14.20, I'll let you know if the issue persists.

dasterintempus commented 7 years ago

0.14.20 has had one of our 4 worker processes deadlock already, here is it's threaddump. I expect the others to deadlock shortly. threaddump_14.20.txt

repeatedly commented 7 years ago

Thanks for the testing. Could you paste your configuration and traffic info? I want to reproduce the issue on my environment.

GLStephen commented 7 years ago

Greetings we are seeing what appears to be deadlocks like these. We're processing a few hundred million events a day on 4 servers. What can I do to help with diagnostics? These are all 14.20 and a couple of the servers (though they have identical gems and versions) run well, while others have issues.

dasterintempus commented 7 years ago

I'm waiting on some corporate approval on how much information I can share. What I can say right now is that our config looks generally like this, and it's operating on messages originally received from syslog.

<match some.tag>
    @type file
    append true
    format custom_format
    path /some/path/${host}/%Y-%m-%d/${tag[3]}_%H
    <buffer tag,host,time>
        path "/other/path/buffer/"

        flush_mode interval
        flush_interval 60s
        flush_thread_count 2

        chunk_limit_size 64MB
        total_limit_size 50GB

        timekey 1h
        timekey_wait 300
    </buffer>
</match>
repeatedly commented 7 years ago

Thanks for the info! I will try to reproduce it on my environment.

repeatedly commented 7 years ago

Hmm... I use the modified version of scottbuettner's conf, but I still can't reproduce the problem on my Mac and Ubuntu 16.04. Here is my test set:

<source>
  @type forward
</source>

<source>
  @type forward
  port 24225
</source>

# To increase lock / unlock, add test field.
<match test.tag>
  @type file
  append true
  format json
  path /home/repeatedly/tmp/deadlock/log/${host}/${test}/%Y-%m-%d/${tag[1]}_%H
  <buffer tag,host,test,time>
    path /home/repeatedly/tmp/deadlock/buffer/
    flush_mode interval
    flush_interval 60s
    flush_thread_count 2
    chunk_limit_size 32MB
    total_limit_size 50GB
    timekey 1h
    timekey_wait 300
    flush_at_shutdown true
  </buffer>
</match>
require 'fluent-logger'
require 'json'

port = ARGV[0].nil? ? 24224 : Integer(ARGV[0])
log = Fluent::Logger::FluentLogger.new(nil, :host=>'localhost', :port => port)

msgs = []
# hoge.json contains the number of nginx logs. Different content is ok.
File.read('hoge.json').each_line { |l|
  m = JSON.parse(l)
  m['host'] = "host#{rand(50)}"
  m['test'] = "test#{rand(10)}"
  msgs << m
}

i = 0
size = msgs.size

loop {
  (0..100000).each { |i|
    msg = msgs[i % size]
    msg['num'] = i
    log.post("test.tag", msg)
    sleep 0.0001
  }
  sleep 1
}

I run 2 or 3 clients in parallel but the problem is not reproduced in 12 hours. Does anyone provide reproducible step for this problem locally?

dasterintempus commented 7 years ago

@repeatedly Can you track how many messages you're actually sending with that? We're receiving somewhere around 3000/second in the system that fails, and it usually does so within a few hours.

I don't think it's relevant, but we're running on RHEL/CentOS 6, rather than Ubuntu. td-agent bundles the ruby interpreter so I don't think it would be much difference there, but I thought I'd mention it.

hfinucane commented 7 years ago

Reporter here, also running CentOS 6.

repeatedly commented 7 years ago

I setup CentOS 6 box and restarted reproduce step with 3000msgs/second.

repeatedly commented 7 years ago

Hmm... I can't still reproduce this problem on CentOS 6. I use ruby 2.1.10 or ruby 2.4.1 with fluentd v0.14.21 and run fluentd / logger in 5 days. Does anyone provide the more detailed information, e.g. hardward spec?

And if you hit the problem again, could you try perf or pid2line.rb? https://docs.fluentd.org/v0.12/articles/trouble-shooting#high-cpu-usage-issue

In addition, ruby 2.4 has better deadlock detection. So does anyone try above configuration with ruby 2.4 on reproducible node? I want to know my setup/configuration can reproduce the problem or not.

hfinucane commented 7 years ago

t2.large in AWS- it was never doing much throughput, cpu usage was pretty minimal, even during bursts.

GLStephen commented 6 years ago

More deadlock info:

Sigdump at 2017-10-03 15:03:53 +0000 process 887 (/usr/sbin/td-agent)
  Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
      /opt/td-agent/embedded/bin/fluentd:22:in `load'
      /opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
      /usr/sbin/td-agent:7:in `load'
      /usr/sbin/td-agent:7:in `<main>'
  Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
  Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
  GC stat:
      count: 60
      heap_allocated_pages: 419
      heap_sorted_length: 419
      heap_allocatable_pages: 0
      heap_available_slots: 170773
      heap_live_slots: 168837
      heap_free_slots: 1936
      heap_final_slots: 0
      heap_marked_slots: 104339
      heap_eden_pages: 419
      heap_tomb_pages: 0
      total_allocated_pages: 419
      total_freed_pages: 0
      total_allocated_objects: 1696085
      total_freed_objects: 1527248
      malloc_increase_bytes: 2865440
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 52
      major_gc_count: 8
      remembered_wb_unprotected_objects: 15144
      remembered_wb_unprotected_objects_limit: 20242
      old_objects: 85273
      old_objects_limit: 156152
      oldmalloc_increase_bytes: 4218592
      oldmalloc_increase_bytes_limit: 16777216
  Built-in objects:
   170,773: TOTAL
    69,874: T_STRING
    24,039: T_ARRAY
    23,536: T_IMEMO
    16,273: T_DATA
    11,641: T_HASH
     7,488: T_FILE
     5,336: T_STRUCT
     4,957: T_OBJECT
     2,525: T_CLASS
     1,860: FREE
     1,392: T_NODE
       576: T_REGEXP
       505: T_BIGNUM
       320: T_ICLASS
       235: T_MODULE
        72: T_MATCH
        69: T_SYMBOL
        65: T_RATIONAL
         9: T_FLOAT
         1: T_COMPLEX
  All objects:
    61,302: String
    15,786: Array
     9,523: Hash
     9,341: Time
     5,898: File
     4,749: Fluent::Plugin::Buffer::Metadata
     3,014: Thread::Mutex
     2,947: Fluent::Plugin::Buffer::FileChunk
     2,376: Proc
     1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
     1,463: Class
       677: Gem::Requirement
       576: Regexp
       478: Gem::Dependency
       307: MessagePack::Buffer
       292: Integer
       287: MessagePack::Packer
       260: Range
       235: Module
       214: Gem::Version
       101: Encoding
        94: MatchData
        91: Gem::Specification
        91: Gem::StubSpecification::StubLine
        91: Gem::StubSpecification
        73: Fluent::MultiEventStream
        69: Symbol
        60: Rational
        57: Fluent::Config::ConfigureProxy
        44: Coolio::IO::Watcher
        27: Thread::Backtrace
        22: IO::Buffer
        22: TCPSocket
        21: Fluent::PluginHelper::Server::TCPCallbackSocket
        21: Fluent::PluginHelper::Server::EventHandler::TCPServer
        20: MessagePack::Unpacker
        20: File::Stat
        19: Thread
        17: OptionParser::Switch::RequiredArgument
        15: Fluent::Config::Section
        13: Fluent::Config::Element
        13: OptionParser::Switch::NoArgument
         9: Float
         8: Fluent::Plugin::Output::FlushThreadState
         8: Fluent::Registry
         7: Method
         7: IO
         7: ThreadError
         6: Fluent::Plugin::Base::State
         6: OptionParser::OptionMap
         4: Object
         3: EOFError
         3: TCPServer
         3: Fluent::PluginHelper::EventLoop::DefaultWatcher
         3: ThreadGroup
         3: Monitor
         3: Coolio::Loop
         3: OptionParser::List
         3: IRB::Notifier::LeveledNotifier
         3: JSON::Ext::Generator::State
         2: Fluent::MessagePackEventStream
         2: Fluent::PluginHelper::Server::ServerInfo
         2: Fluent::Log
         2: Logger::Formatter
         2: ServerEngine::DaemonLogger
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: ThreadSafe::Cache
         2: MessagePack::Factory
         2: OpenSSL::PKey::DH
         2: UnboundMethod
         1: Fluent::PluginHelper::Server::EventHandler::UDPServer
         1: UDPSocket
         1: Coolio::TCPServer
         1: WEBrick::HTTPServer::MountTable
         1: Thread::SizedQueue
         1: WEBrick::HTTPServer
         1: WEBrick::Log
         1: DRb::DRbMessage
         1: DRb::DRbTCPSocket
         1: DRb::DRbServer
         1: Fluent::PluginHelper::Timer::TimerWatcher
         1: Fluent::GlobMatchPattern
         1: Fluent::EventRouter::Rule
         1: Fluent::Plugin::ForwardOutput::Node
         1: Fluent::Plugin::ForwardOutput::FailureDetector
         1: Fluent::Plugin::SecondaryFileOutput
         1: ArgumentError
         1: DRb::DRbIdConv
         1: Fluent::EngineClass
         1: Fluent::Compat::NullOutputChain
         1: Fluent::Plugin::ForwardInput
         1: OpenSSL::X509::Store
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Fluent::Plugin::MonitorAgentInput
         1: StringScanner
         1: Fluent::LogDeviceIO
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: OptionParser
         1: WEBrick::HTTPVersion
         1: Fluent::EventRouter::MatchCache
         1: Fluent::EventRouter
         1: Fluent::NoMatchMatch
         1: Fluent::RootAgent
         1: Fluent::SystemConfig
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: IRB::Notifier::NoMsgNotifier
         1: Fluent::PluginLogger
         1: Fluent::Plugin::DebugAgentInput
         1: Fluent::Plugin::ForwardOutput
         1: Set
         1: OptionParser::CompletingHash
         1: Gem::Platform
         1: #<Class:0x007fb4ac4daf18>
         1: Gem::PathSupport
         1: #<Class:0x007fb4ac536b38>
         1: fatal
         1: Data
         1: Fluent::Plugin::FileBuffer
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
  String 4,271,107 bytes
   Array 1 elements
    Hash 223 pairs
Sigdump at 2017-10-03 15:04:13 +0000 process 887 (/usr/sbin/td-agent)
  Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
      /opt/td-agent/embedded/bin/fluentd:22:in `load'
      /opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
      /usr/sbin/td-agent:7:in `load'
      /usr/sbin/td-agent:7:in `<main>'
  Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
  Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
  GC stat:
      count: 60
      heap_allocated_pages: 419
      heap_sorted_length: 419
      heap_allocatable_pages: 0
      heap_available_slots: 170773
      heap_live_slots: 157455
      heap_free_slots: 13318
      heap_final_slots: 0
      heap_marked_slots: 104339
      heap_eden_pages: 419
      heap_tomb_pages: 0
      total_allocated_pages: 419
      total_freed_pages: 0
      total_allocated_objects: 1698768
      total_freed_objects: 1541313
      malloc_increase_bytes: 538912
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 52
      major_gc_count: 8
      remembered_wb_unprotected_objects: 15144
      remembered_wb_unprotected_objects_limit: 20242
      old_objects: 85273
      old_objects_limit: 156152
      oldmalloc_increase_bytes: 1892064
      oldmalloc_increase_bytes_limit: 16777216
  Built-in objects:
   170,773: TOTAL
    62,795: T_STRING
    23,579: T_IMEMO
    22,634: T_ARRAY
    15,655: T_DATA
    13,242: FREE
     9,860: T_HASH
     7,483: T_FILE
     5,028: T_STRUCT
     4,900: T_OBJECT
     2,525: T_CLASS
     1,392: T_NODE
       576: T_REGEXP
       320: T_ICLASS
       292: T_BIGNUM
       235: T_MODULE
       118: T_MATCH
        69: T_SYMBOL
        60: T_RATIONAL
         9: T_FLOAT
         1: T_COMPLEX
  All objects:
    62,969: String
    16,141: Array
     9,527: Hash
     9,343: Time
     5,899: File
     4,749: Fluent::Plugin::Buffer::Metadata
     3,015: Thread::Mutex
     2,947: Fluent::Plugin::Buffer::FileChunk
     2,377: Proc
     1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
     1,463: Class
       677: Gem::Requirement
       576: Regexp
       478: Gem::Dependency
       307: MessagePack::Buffer
       292: Integer
       287: MessagePack::Packer
       260: Range
       235: Module
       214: Gem::Version
       142: MatchData
       101: Encoding
        91: Gem::Specification
        91: Gem::StubSpecification::StubLine
        91: Gem::StubSpecification
        73: Fluent::MultiEventStream
        69: Symbol
        60: Rational
        57: Fluent::Config::ConfigureProxy
        44: Coolio::IO::Watcher
        44: Thread::Backtrace
        22: IO::Buffer
        22: TCPSocket
        21: Fluent::PluginHelper::Server::TCPCallbackSocket
        21: Fluent::PluginHelper::Server::EventHandler::TCPServer
        20: MessagePack::Unpacker
        20: File::Stat
        19: Thread
        17: OptionParser::Switch::RequiredArgument
        15: Fluent::Config::Section
        13: Fluent::Config::Element
        13: OptionParser::Switch::NoArgument
         9: Float
         8: ThreadError
         8: Fluent::Plugin::Output::FlushThreadState
         8: Fluent::Registry
         7: Method
         7: IO
         6: Fluent::Plugin::Base::State
         6: OptionParser::OptionMap
         4: Object
         3: EOFError
         3: TCPServer
         3: Fluent::PluginHelper::EventLoop::DefaultWatcher
         3: ThreadGroup
         3: Monitor
         3: Coolio::Loop
         3: OptionParser::List
         3: IRB::Notifier::LeveledNotifier
         3: JSON::Ext::Generator::State
         2: Fluent::MessagePackEventStream
         2: Fluent::PluginHelper::Server::ServerInfo
         2: Fluent::Log
         2: Logger::Formatter
         2: ServerEngine::DaemonLogger
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: ThreadSafe::Cache
         2: MessagePack::Factory
         2: OpenSSL::PKey::DH
         2: UnboundMethod
         1: Fluent::PluginHelper::Server::EventHandler::UDPServer
         1: UDPSocket
         1: Coolio::TCPServer
         1: WEBrick::HTTPServer::MountTable
         1: Thread::SizedQueue
         1: WEBrick::HTTPServer
         1: WEBrick::Log
         1: DRb::DRbMessage
         1: DRb::DRbTCPSocket
         1: DRb::DRbServer
         1: Fluent::PluginHelper::Timer::TimerWatcher
         1: Fluent::GlobMatchPattern
         1: Fluent::EventRouter::Rule
         1: Fluent::Plugin::ForwardOutput::Node
         1: Fluent::Plugin::ForwardOutput::FailureDetector
         1: Fluent::Plugin::SecondaryFileOutput
         1: ArgumentError
         1: DRb::DRbIdConv
         1: Fluent::EngineClass
         1: Fluent::Compat::NullOutputChain
         1: Fluent::Plugin::ForwardInput
         1: OpenSSL::X509::Store
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Fluent::Plugin::MonitorAgentInput
         1: StringScanner
         1: Fluent::LogDeviceIO
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: OptionParser
         1: WEBrick::HTTPVersion
         1: Fluent::EventRouter::MatchCache
         1: Fluent::EventRouter
         1: Fluent::NoMatchMatch
         1: Fluent::RootAgent
         1: Fluent::SystemConfig
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: IRB::Notifier::NoMsgNotifier
         1: Fluent::PluginLogger
         1: Fluent::Plugin::DebugAgentInput
         1: Fluent::Plugin::ForwardOutput
         1: Set
         1: OptionParser::CompletingHash
         1: Gem::Platform
         1: #<Class:0x007fb4ac4daf18>
         1: Gem::PathSupport
         1: #<Class:0x007fb4ac536b38>
         1: fatal
         1: Data
         1: Fluent::Plugin::FileBuffer
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
  String 4,322,053 bytes
   Array 2 elements
    Hash 223 pairs
Sigdump at 2017-10-03 15:05:17 +0000 process 887 (/usr/sbin/td-agent)
  Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
      /opt/td-agent/embedded/bin/fluentd:22:in `load'
      /opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
      /usr/sbin/td-agent:7:in `load'
      /usr/sbin/td-agent:7:in `<main>'
  Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
  Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
  GC stat:
      count: 60
      heap_allocated_pages: 419
      heap_sorted_length: 419
      heap_allocatable_pages: 0
      heap_available_slots: 170773
      heap_live_slots: 159739
      heap_free_slots: 11034
      heap_final_slots: 0
      heap_marked_slots: 104339
      heap_eden_pages: 419
      heap_tomb_pages: 0
      total_allocated_pages: 419
      total_freed_pages: 0
      total_allocated_objects: 1701414
      total_freed_objects: 1541675
      malloc_increase_bytes: 622512
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 52
      major_gc_count: 8
      remembered_wb_unprotected_objects: 15144
      remembered_wb_unprotected_objects_limit: 20242
      old_objects: 85273
      old_objects_limit: 156152
      oldmalloc_increase_bytes: 1975664
      oldmalloc_increase_bytes_limit: 16777216
  Built-in objects:
   170,773: TOTAL
    64,449: T_STRING
    23,742: T_IMEMO
    23,030: T_ARRAY
    15,674: T_DATA
    10,958: FREE
     9,863: T_HASH
     7,484: T_FILE
     5,028: T_STRUCT
     4,900: T_OBJECT
     2,525: T_CLASS
     1,392: T_NODE
       576: T_REGEXP
       320: T_ICLASS
       292: T_BIGNUM
       235: T_MODULE
       166: T_MATCH
        69: T_SYMBOL
        60: T_RATIONAL
         9: T_FLOAT
         1: T_COMPLEX
  All objects:
    64,623: String
    16,535: Array
     9,530: Hash
     9,344: Time
     5,900: File
     4,749: Fluent::Plugin::Buffer::Metadata
     3,016: Thread::Mutex
     2,947: Fluent::Plugin::Buffer::FileChunk
     2,378: Proc
     1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
     1,463: Class
       677: Gem::Requirement
       576: Regexp
       478: Gem::Dependency
       307: MessagePack::Buffer
       292: Integer
       287: MessagePack::Packer
       260: Range
       235: Module
       214: Gem::Version
       190: MatchData
       101: Encoding
        91: Gem::Specification
        91: Gem::StubSpecification::StubLine
        91: Gem::StubSpecification
        73: Fluent::MultiEventStream
        69: Symbol
        60: Thread::Backtrace
        60: Rational
        57: Fluent::Config::ConfigureProxy
        44: Coolio::IO::Watcher
        22: IO::Buffer
        22: TCPSocket
        21: Fluent::PluginHelper::Server::TCPCallbackSocket
        21: Fluent::PluginHelper::Server::EventHandler::TCPServer
        20: MessagePack::Unpacker
        20: File::Stat
        19: Thread
        17: OptionParser::Switch::RequiredArgument
        15: Fluent::Config::Section
        13: Fluent::Config::Element
        13: OptionParser::Switch::NoArgument
         9: Float
         8: ThreadError
         8: Fluent::Plugin::Output::FlushThreadState
         8: Fluent::Registry
         7: Method
         7: IO
         6: Fluent::Plugin::Base::State
         6: OptionParser::OptionMap
         4: Object
         3: EOFError
         3: TCPServer
         3: Fluent::PluginHelper::EventLoop::DefaultWatcher
         3: ThreadGroup
         3: Monitor
         3: Coolio::Loop
         3: OptionParser::List
         3: IRB::Notifier::LeveledNotifier
         3: JSON::Ext::Generator::State
         2: Fluent::MessagePackEventStream
         2: Fluent::PluginHelper::Server::ServerInfo
         2: Fluent::Log
         2: Logger::Formatter
         2: ServerEngine::DaemonLogger
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: ThreadSafe::Cache
         2: MessagePack::Factory
         2: OpenSSL::PKey::DH
         2: UnboundMethod
         1: Fluent::PluginHelper::Server::EventHandler::UDPServer
         1: UDPSocket
         1: Coolio::TCPServer
         1: WEBrick::HTTPServer::MountTable
         1: Thread::SizedQueue
         1: WEBrick::HTTPServer
         1: WEBrick::Log
         1: DRb::DRbMessage
         1: DRb::DRbTCPSocket
         1: DRb::DRbServer
         1: Fluent::PluginHelper::Timer::TimerWatcher
         1: Fluent::GlobMatchPattern
         1: Fluent::EventRouter::Rule
         1: Fluent::Plugin::ForwardOutput::Node
         1: Fluent::Plugin::ForwardOutput::FailureDetector
         1: Fluent::Plugin::SecondaryFileOutput
         1: ArgumentError
         1: DRb::DRbIdConv
         1: Fluent::EngineClass
         1: Fluent::Compat::NullOutputChain
         1: Fluent::Plugin::ForwardInput
         1: OpenSSL::X509::Store
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Fluent::Plugin::MonitorAgentInput
         1: StringScanner
         1: Fluent::LogDeviceIO
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: OptionParser
         1: WEBrick::HTTPVersion
         1: Fluent::EventRouter::MatchCache
         1: Fluent::EventRouter
         1: Fluent::NoMatchMatch
         1: Fluent::RootAgent
         1: Fluent::SystemConfig
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: IRB::Notifier::NoMsgNotifier
         1: Fluent::PluginLogger
         1: Fluent::Plugin::DebugAgentInput
         1: Fluent::Plugin::ForwardOutput
         1: Set
         1: OptionParser::CompletingHash
         1: Gem::Platform
         1: #<Class:0x007fb4ac4daf18>
         1: Gem::PathSupport
         1: #<Class:0x007fb4ac536b38>
         1: fatal
         1: Data
         1: Fluent::Plugin::FileBuffer
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
  String 4,372,518 bytes
   Array 2 elements
    Hash 223 pairs
Sigdump at 2017-10-03 15:05:21 +0000 process 887 (/usr/sbin/td-agent)
  Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
      /opt/td-agent/embedded/bin/fluentd:22:in `load'
      /opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
      /usr/sbin/td-agent:7:in `load'
      /usr/sbin/td-agent:7:in `<main>'
  Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
  Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
  GC stat:
      count: 60
      heap_allocated_pages: 419
      heap_sorted_length: 419
      heap_allocatable_pages: 0
      heap_available_slots: 170773
      heap_live_slots: 161963
      heap_free_slots: 8810
      heap_final_slots: 0
      heap_marked_slots: 104339
      heap_eden_pages: 419
      heap_tomb_pages: 0
      total_allocated_pages: 419
      total_freed_pages: 0
      total_allocated_objects: 1704000
      total_freed_objects: 1542037
      malloc_increase_bytes: 706112
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 52
      major_gc_count: 8
      remembered_wb_unprotected_objects: 15144
      remembered_wb_unprotected_objects_limit: 20242
      old_objects: 85273
      old_objects_limit: 156152
      oldmalloc_increase_bytes: 2059264
      oldmalloc_increase_bytes_limit: 16777216
  Built-in objects:
   170,773: TOTAL
    66,103: T_STRING
    23,905: T_IMEMO
    23,366: T_ARRAY
    15,693: T_DATA
     9,866: T_HASH
     8,734: FREE
     7,485: T_FILE
     5,028: T_STRUCT
     4,900: T_OBJECT
     2,525: T_CLASS
     1,392: T_NODE
       576: T_REGEXP
       320: T_ICLASS
       292: T_BIGNUM
       235: T_MODULE
       214: T_MATCH
        69: T_SYMBOL
        60: T_RATIONAL
         9: T_FLOAT
         1: T_COMPLEX
  All objects:
    66,277: String
    16,869: Array
     9,533: Hash
     9,345: Time
     5,901: File
     4,749: Fluent::Plugin::Buffer::Metadata
     3,017: Thread::Mutex
     2,947: Fluent::Plugin::Buffer::FileChunk
     2,379: Proc
     1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
     1,463: Class
       677: Gem::Requirement
       576: Regexp
       478: Gem::Dependency
       307: MessagePack::Buffer
       292: Integer
       287: MessagePack::Packer
       260: Range
       238: MatchData
       235: Module
       214: Gem::Version
       101: Encoding
        91: Gem::Specification
        91: Gem::StubSpecification::StubLine
        91: Gem::StubSpecification
        76: Thread::Backtrace
        73: Fluent::MultiEventStream
        69: Symbol
        60: Rational
        57: Fluent::Config::ConfigureProxy
        44: Coolio::IO::Watcher
        22: IO::Buffer
        22: TCPSocket
        21: Fluent::PluginHelper::Server::TCPCallbackSocket
        21: Fluent::PluginHelper::Server::EventHandler::TCPServer
        20: MessagePack::Unpacker
        20: File::Stat
        19: Thread
        17: OptionParser::Switch::RequiredArgument
        15: Fluent::Config::Section
        13: Fluent::Config::Element
        13: OptionParser::Switch::NoArgument
         9: Float
         8: ThreadError
         8: Fluent::Plugin::Output::FlushThreadState
         8: Fluent::Registry
         7: Method
         7: IO
         6: Fluent::Plugin::Base::State
         6: OptionParser::OptionMap
         4: Object
         3: EOFError
         3: TCPServer
         3: Fluent::PluginHelper::EventLoop::DefaultWatcher
         3: ThreadGroup
         3: Monitor
         3: Coolio::Loop
         3: OptionParser::List
         3: IRB::Notifier::LeveledNotifier
         3: JSON::Ext::Generator::State
         2: Fluent::MessagePackEventStream
         2: Fluent::PluginHelper::Server::ServerInfo
         2: Fluent::Log
         2: Logger::Formatter
         2: ServerEngine::DaemonLogger
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: ThreadSafe::Cache
         2: MessagePack::Factory
         2: OpenSSL::PKey::DH
         2: UnboundMethod
         1: Fluent::PluginHelper::Server::EventHandler::UDPServer
         1: UDPSocket
         1: Coolio::TCPServer
         1: WEBrick::HTTPServer::MountTable
         1: Thread::SizedQueue
         1: WEBrick::HTTPServer
         1: WEBrick::Log
         1: DRb::DRbMessage
         1: DRb::DRbTCPSocket
         1: DRb::DRbServer
         1: Fluent::PluginHelper::Timer::TimerWatcher
         1: Fluent::GlobMatchPattern
         1: Fluent::EventRouter::Rule
         1: Fluent::Plugin::ForwardOutput::Node
         1: Fluent::Plugin::ForwardOutput::FailureDetector
         1: Fluent::Plugin::SecondaryFileOutput
         1: ArgumentError
         1: DRb::DRbIdConv
         1: Fluent::EngineClass
         1: Fluent::Compat::NullOutputChain
         1: Fluent::Plugin::ForwardInput
         1: OpenSSL::X509::Store
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Fluent::Plugin::MonitorAgentInput
         1: StringScanner
         1: Fluent::LogDeviceIO
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: OptionParser
         1: WEBrick::HTTPVersion
         1: Fluent::EventRouter::MatchCache
         1: Fluent::EventRouter
         1: Fluent::NoMatchMatch
         1: Fluent::RootAgent
         1: Fluent::SystemConfig
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: IRB::Notifier::NoMsgNotifier
         1: Fluent::PluginLogger
         1: Fluent::Plugin::DebugAgentInput
         1: Fluent::Plugin::ForwardOutput
         1: Set
         1: OptionParser::CompletingHash
         1: Gem::Platform
         1: #<Class:0x007fb4ac4daf18>
         1: Gem::PathSupport
         1: #<Class:0x007fb4ac536b38>
         1: fatal
         1: Data
         1: Fluent::Plugin::FileBuffer
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
  String 4,422,977 bytes
   Array 2 elements
    Hash 223 pairs
Sigdump at 2017-10-03 15:06:01 +0000 process 887 (/usr/sbin/td-agent)
  Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
      /opt/td-agent/embedded/bin/fluentd:22:in `load'
      /opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
      /usr/sbin/td-agent:7:in `load'
      /usr/sbin/td-agent:7:in `<main>'
  Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
  Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
  GC stat:
      count: 60
      heap_allocated_pages: 419
      heap_sorted_length: 419
      heap_allocatable_pages: 0
      heap_available_slots: 170773
      heap_live_slots: 164243
      heap_free_slots: 6530
      heap_final_slots: 0
      heap_marked_slots: 104339
      heap_eden_pages: 419
      heap_tomb_pages: 0
      total_allocated_pages: 419
      total_freed_pages: 0
      total_allocated_objects: 1706643
      total_freed_objects: 1542400
      malloc_increase_bytes: 791136
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 52
      major_gc_count: 8
      remembered_wb_unprotected_objects: 15144
      remembered_wb_unprotected_objects_limit: 20242
      old_objects: 85273
      old_objects_limit: 156152
      oldmalloc_increase_bytes: 2144288
      oldmalloc_increase_bytes_limit: 16777216
  Built-in objects:
   170,773: TOTAL
    67,768: T_STRING
    24,068: T_IMEMO
    23,743: T_ARRAY
    15,714: T_DATA
     9,870: T_HASH
     7,486: T_FILE
     6,454: FREE
     5,028: T_STRUCT
     4,901: T_OBJECT
     2,525: T_CLASS
     1,392: T_NODE
       576: T_REGEXP
       320: T_ICLASS
       292: T_BIGNUM
       262: T_MATCH
       235: T_MODULE
        69: T_SYMBOL
        60: T_RATIONAL
         9: T_FLOAT
         1: T_COMPLEX
  All objects:
    67,942: String
    17,244: Array
     9,537: Hash
     9,347: Time
     5,902: File
     4,749: Fluent::Plugin::Buffer::Metadata
     3,018: Thread::Mutex
     2,947: Fluent::Plugin::Buffer::FileChunk
     2,380: Proc
     1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
     1,463: Class
       677: Gem::Requirement
       576: Regexp
       478: Gem::Dependency
       307: MessagePack::Buffer
       292: Integer
       287: MessagePack::Packer
       286: MatchData
       260: Range
       235: Module
       214: Gem::Version
       101: Encoding
        93: Thread::Backtrace
        91: Gem::Specification
        91: Gem::StubSpecification::StubLine
        91: Gem::StubSpecification
        73: Fluent::MultiEventStream
        69: Symbol
        60: Rational
        57: Fluent::Config::ConfigureProxy
        44: Coolio::IO::Watcher
        22: IO::Buffer
        22: TCPSocket
        21: Fluent::PluginHelper::Server::TCPCallbackSocket
        21: Fluent::PluginHelper::Server::EventHandler::TCPServer
        20: MessagePack::Unpacker
        20: File::Stat
        19: Thread
        17: OptionParser::Switch::RequiredArgument
        15: Fluent::Config::Section
        13: Fluent::Config::Element
        13: OptionParser::Switch::NoArgument
         9: ThreadError
         9: Float
         8: Fluent::Plugin::Output::FlushThreadState
         8: Fluent::Registry
         7: Method
         7: IO
         6: Fluent::Plugin::Base::State
         6: OptionParser::OptionMap
         4: Object
         3: EOFError
         3: TCPServer
         3: Fluent::PluginHelper::EventLoop::DefaultWatcher
         3: ThreadGroup
         3: Monitor
         3: Coolio::Loop
         3: OptionParser::List
         3: IRB::Notifier::LeveledNotifier
         3: JSON::Ext::Generator::State
         2: Fluent::MessagePackEventStream
         2: Fluent::PluginHelper::Server::ServerInfo
         2: Fluent::Log
         2: Logger::Formatter
         2: ServerEngine::DaemonLogger
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: ThreadSafe::Cache
         2: MessagePack::Factory
         2: OpenSSL::PKey::DH
         2: UnboundMethod
         1: Fluent::PluginHelper::Server::EventHandler::UDPServer
         1: UDPSocket
         1: Coolio::TCPServer
         1: WEBrick::HTTPServer::MountTable
         1: Thread::SizedQueue
         1: WEBrick::HTTPServer
         1: WEBrick::Log
         1: DRb::DRbMessage
         1: DRb::DRbTCPSocket
         1: DRb::DRbServer
         1: Fluent::PluginHelper::Timer::TimerWatcher
         1: Fluent::GlobMatchPattern
         1: Fluent::EventRouter::Rule
         1: Fluent::Plugin::ForwardOutput::Node
         1: Fluent::Plugin::ForwardOutput::FailureDetector
         1: Fluent::Plugin::SecondaryFileOutput
         1: ArgumentError
         1: DRb::DRbIdConv
         1: Fluent::EngineClass
         1: Fluent::Compat::NullOutputChain
         1: Fluent::Plugin::ForwardInput
         1: OpenSSL::X509::Store
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Fluent::Plugin::MonitorAgentInput
         1: StringScanner
         1: Fluent::LogDeviceIO
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: OptionParser
         1: WEBrick::HTTPVersion
         1: Fluent::EventRouter::MatchCache
         1: Fluent::EventRouter
         1: Fluent::NoMatchMatch
         1: Fluent::RootAgent
         1: Fluent::SystemConfig
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: IRB::Notifier::NoMsgNotifier
         1: Fluent::PluginLogger
         1: Fluent::Plugin::DebugAgentInput
         1: Fluent::Plugin::ForwardOutput
         1: Set
         1: OptionParser::CompletingHash
         1: Gem::Platform
         1: #<Class:0x007fb4ac4daf18>
         1: Gem::PathSupport
         1: #<Class:0x007fb4ac536b38>
         1: fatal
         1: Data
         1: Fluent::Plugin::FileBuffer
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
  String 4,473,868 bytes
   Array 2 elements
    Hash 223 pairs
Sigdump at 2017-10-03 15:06:23 +0000 process 887 (/usr/sbin/td-agent)
  Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
      /opt/td-agent/embedded/bin/fluentd:22:in `load'
      /opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
      /usr/sbin/td-agent:7:in `load'
      /usr/sbin/td-agent:7:in `<main>'
  Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
  Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
  GC stat:
      count: 60
      heap_allocated_pages: 419
      heap_sorted_length: 419
      heap_allocatable_pages: 0
      heap_available_slots: 170773
      heap_live_slots: 166485
      heap_free_slots: 4288
      heap_final_slots: 0
      heap_marked_slots: 104339
      heap_eden_pages: 419
      heap_tomb_pages: 0
      total_allocated_pages: 419
      total_freed_pages: 0
      total_allocated_objects: 1709247
      total_freed_objects: 1542762
      malloc_increase_bytes: 874736
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 52
      major_gc_count: 8
      remembered_wb_unprotected_objects: 15144
      remembered_wb_unprotected_objects_limit: 20242
      old_objects: 85273
      old_objects_limit: 156152
      oldmalloc_increase_bytes: 2227888
      oldmalloc_increase_bytes_limit: 16777216
  Built-in objects:
   170,773: TOTAL
    69,422: T_STRING
    24,231: T_IMEMO
    24,097: T_ARRAY
    15,733: T_DATA
     9,873: T_HASH
     7,487: T_FILE
     5,028: T_STRUCT
     4,901: T_OBJECT
     4,212: FREE
     2,525: T_CLASS
     1,392: T_NODE
       576: T_REGEXP
       320: T_ICLASS
       310: T_MATCH
       292: T_BIGNUM
       235: T_MODULE
        69: T_SYMBOL
        60: T_RATIONAL
         9: T_FLOAT
         1: T_COMPLEX
  All objects:
    69,596: String
    17,596: Array
     9,540: Hash
     9,348: Time
     5,903: File
     4,749: Fluent::Plugin::Buffer::Metadata
     3,019: Thread::Mutex
     2,947: Fluent::Plugin::Buffer::FileChunk
     2,381: Proc
     1,551: Fluent::PluginHelper::Socket::WrappedSocket::TCP
     1,463: Class
       677: Gem::Requirement
       576: Regexp
       478: Gem::Dependency
       334: MatchData
       307: MessagePack::Buffer
       292: Integer
       287: MessagePack::Packer
       260: Range
       235: Module
       214: Gem::Version
       109: Thread::Backtrace
       101: Encoding
        91: Gem::Specification
        91: Gem::StubSpecification::StubLine
        91: Gem::StubSpecification
        73: Fluent::MultiEventStream
        69: Symbol
        60: Rational
        57: Fluent::Config::ConfigureProxy
        44: Coolio::IO::Watcher
        22: IO::Buffer
        22: TCPSocket
        21: Fluent::PluginHelper::Server::TCPCallbackSocket
        21: Fluent::PluginHelper::Server::EventHandler::TCPServer
        20: MessagePack::Unpacker
        20: File::Stat
        19: Thread
        17: OptionParser::Switch::RequiredArgument
        15: Fluent::Config::Section
        13: Fluent::Config::Element
        13: OptionParser::Switch::NoArgument
         9: ThreadError
         9: Float
         8: Fluent::Plugin::Output::FlushThreadState
         8: Fluent::Registry
         7: Method
         7: IO
         6: Fluent::Plugin::Base::State
         6: OptionParser::OptionMap
         4: Object
         3: EOFError
         3: TCPServer
         3: Fluent::PluginHelper::EventLoop::DefaultWatcher
         3: ThreadGroup
         3: Monitor
         3: Coolio::Loop
         3: OptionParser::List
         3: IRB::Notifier::LeveledNotifier
         3: JSON::Ext::Generator::State
         2: Fluent::MessagePackEventStream
         2: Fluent::PluginHelper::Server::ServerInfo
         2: Fluent::Log
         2: Logger::Formatter
         2: ServerEngine::DaemonLogger
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: ThreadSafe::Cache
         2: MessagePack::Factory
         2: OpenSSL::PKey::DH
         2: UnboundMethod
         1: Fluent::PluginHelper::Server::EventHandler::UDPServer
         1: UDPSocket
         1: Coolio::TCPServer
         1: WEBrick::HTTPServer::MountTable
         1: Thread::SizedQueue
         1: WEBrick::HTTPServer
         1: WEBrick::Log
         1: DRb::DRbMessage
         1: DRb::DRbTCPSocket
         1: DRb::DRbServer
         1: Fluent::PluginHelper::Timer::TimerWatcher
         1: Fluent::GlobMatchPattern
         1: Fluent::EventRouter::Rule
         1: Fluent::Plugin::ForwardOutput::Node
         1: Fluent::Plugin::ForwardOutput::FailureDetector
         1: Fluent::Plugin::SecondaryFileOutput
         1: ArgumentError
         1: DRb::DRbIdConv
         1: Fluent::EngineClass
         1: Fluent::Compat::NullOutputChain
         1: Fluent::Plugin::ForwardInput
         1: OpenSSL::X509::Store
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Fluent::Plugin::MonitorAgentInput
         1: StringScanner
         1: Fluent::LogDeviceIO
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: OptionParser
         1: WEBrick::HTTPVersion
         1: Fluent::EventRouter::MatchCache
         1: Fluent::EventRouter
         1: Fluent::NoMatchMatch
         1: Fluent::RootAgent
         1: Fluent::SystemConfig
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: IRB::Notifier::NoMsgNotifier
         1: Fluent::PluginLogger
         1: Fluent::Plugin::DebugAgentInput
         1: Fluent::Plugin::ForwardOutput
         1: Set
         1: OptionParser::CompletingHash
         1: Gem::Platform
         1: #<Class:0x007fb4ac4daf18>
         1: Gem::PathSupport
         1: #<Class:0x007fb4ac536b38>
         1: fatal
         1: Data
         1: Fluent::Plugin::FileBuffer
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
  String 4,524,327 bytes
   Array 2 elements
    Hash 223 pairs
Sigdump at 2017-10-03 15:07:54 +0000 process 887 (/usr/sbin/td-agent)
  Thread #<Thread:0x007fb4b35b5f58> status=run priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (3 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:223:in `block (2 levels) in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:158:in `block in lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:140:in `lifecycle'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:226:in `block in shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:248:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:278:in `shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/engine.rb:247:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:772:in `run_engine'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:521:in `block in run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:697:in `main_process'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/supervisor.rb:516:in `run_worker'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/bin/fluentd:5:in `<top (required)>'
      /opt/td-agent/embedded/bin/fluentd:22:in `load'
      /opt/td-agent/embedded/bin/fluentd:22:in `<top (required)>'
      /usr/sbin/td-agent:7:in `load'
      /usr/sbin/td-agent:7:in `<main>'
  Thread #<Thread:0x007fb4a841b978> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841aed8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a841a578> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418fc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a8418408> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3559f50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3566d18> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:395:in `purge_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:934:in `commit_write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1047:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4b3577500> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:369:in `dequeue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1007:in `try_flush'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1268:in `flush_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:420:in `block (2 levels) in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffd90> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:1231:in `enqueue_thread_run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60ffae8> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60fde78> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:991:in `accept_or_shutdown'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:977:in `accept'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1654:in `main_loop'
      /opt/td-agent/embedded/lib/ruby/2.4.0/drb/drb.rb:1500:in `block in run'
  Thread #<Thread:0x007fb4a60fdcc0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb810> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `select'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:171:in `block in start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:33:in `start'
      /opt/td-agent/embedded/lib/ruby/2.4.0/webrick/server.rb:158:in `start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_monitor_agent.rb:253:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a60cb5e0> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:457:in `write_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:207:in `block in write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/buffer.rb:206:in `write'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:896:in `block in handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:805:in `write_guard'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:895:in `handle_stream_with_standard_format'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:799:in `execute_chunking'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:732:in `emit_buffered'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/event_router.rb:96:in `emit_stream'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:300:in `on_message'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:123:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/io.rb:186:in `on_readable'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run_once'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/cool.io-1.5.0/lib/cool.io/loop.rb:88:in `run'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb4a69f4e50> status=sleep priority=0
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `join'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/plugin/output.rb:460:in `before_shutdown'
      /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-0.14.20/lib/fluent/root_agent.rb:235:in `block (3 levels) in shutdown'
  GC stat:
      count: 61
      heap_allocated_pages: 419
      heap_sorted_length: 419
      heap_allocatable_pages: 0
      heap_available_slots: 170773
      heap_live_slots: 167229
      heap_free_slots: 3544
      heap_final_slots: 0
      heap_marked_slots: 107398
      heap_eden_pages: 419
      heap_tomb_pages: 0
      total_allocated_pages: 419
      total_freed_pages: 0
      total_allocated_objects: 1711942
      total_freed_objects: 1544713
      malloc_increase_bytes: 58032
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 53
      major_gc_count: 8
      remembered_wb_unprotected_objects: 15506
      remembered_wb_unprotected_objects_limit: 20242
      old_objects: 86119
      old_objects_limit: 156152
      oldmalloc_increase_bytes: 2268288
      oldmalloc_increase_bytes_limit: 16777216
  Built-in objects:
   170,773: TOTAL
    69,478: T_STRING
    24,394: T_IMEMO
    24,292: T_ARRAY
    15,754: T_DATA
     9,781: T_HASH
     7,488: T_FILE
     5,001: T_STRUCT
     4,881: T_OBJECT
     3,867: FREE
     2,525: T_CLASS
     1,392: T_NODE
       576: T_REGEXP
       358: T_MATCH
       320: T_ICLASS
       292: T_BIGNUM
       235: T_MODULE
        69: T_SYMBOL
        60: T_RATIONAL
         9: T_FLOAT
         1: T_COMPLEX
  All objects:
    47,724: String
     6,013: Time
     5,900: File
     4,647: Array
     4,595: Fluent::Plugin::Buffer::Metadata
     3,016: Thread::Mutex
     2,947: Fluent::Plugin::Buffer::FileChunk
     1,463: Class
       923: Hash
       677: Gem::Requirement
       576: Regexp
       540: Proc
       478: Gem::Dependency
       235: Module
       214: Gem::Version
       101: Encoding
        91: Gem::Specification
        91: Gem::StubSpecification::StubLine
        91: Gem::StubSpecification
        69: Symbol
        59: Rational
        57: Fluent::Config::ConfigureProxy
        46: Range
        44: Coolio::IO::Watcher
        27: MatchData
        22: IO::Buffer
        22: Thread::Backtrace
        21: Fluent::PluginHelper::Server::TCPCallbackSocket
        21: Fluent::PluginHelper::Server::EventHandler::TCPServer
        21: TCPSocket
        18: MessagePack::Buffer
        18: MessagePack::Unpacker
        17: Thread
        17: OptionParser::Switch::RequiredArgument
        15: Fluent::Config::Section
        13: Fluent::Config::Element
        13: OptionParser::Switch::NoArgument
        12: File::Stat
         9: Float
         8: Fluent::Plugin::Output::FlushThreadState
         8: Fluent::Registry
         7: Method
         7: IO
         7: Integer
         6: Fluent::Plugin::Base::State
         6: OptionParser::OptionMap
         4: Object
         3: ThreadError
         3: TCPServer
         3: Fluent::PluginHelper::EventLoop::DefaultWatcher
         3: ThreadGroup
         3: Monitor
         3: Coolio::Loop
         3: OptionParser::List
         3: IRB::Notifier::LeveledNotifier
         3: JSON::Ext::Generator::State
         2: Fluent::MultiEventStream
         2: EOFError
         2: Fluent::PluginHelper::Server::ServerInfo
         2: Fluent::Log
         2: Logger::Formatter
         2: ServerEngine::DaemonLogger
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: ThreadSafe::Cache
         2: MessagePack::Factory
         2: OpenSSL::PKey::DH
         2: UnboundMethod
         1: Fluent::MessagePackEventStream
         1: Fluent::PluginHelper::Server::EventHandler::UDPServer
         1: UDPSocket
         1: Coolio::TCPServer
         1: WEBrick::HTTPServer::MountTable
         1: Thread::SizedQueue
         1: WEBrick::HTTPServer
         1: WEBrick::Log
         1: DRb::DRbMessage
         1: DRb::DRbTCPSocket
         1: DRb::DRbServer
         1: Fluent::PluginHelper::Timer::TimerWatcher
         1: Fluent::GlobMatchPattern
         1: Fluent::EventRouter::Rule
         1: Fluent::Plugin::ForwardOutput::Node
         1: Fluent::Plugin::ForwardOutput::FailureDetector
         1: Fluent::Plugin::SecondaryFileOutput
         1: ArgumentError
         1: DRb::DRbIdConv
         1: Fluent::EngineClass
         1: Fluent::Compat::NullOutputChain
         1: Fluent::Plugin::ForwardInput
         1: OpenSSL::X509::Store
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Fluent::Plugin::MonitorAgentInput
         1: StringScanner
         1: Fluent::LogDeviceIO
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: OptionParser
         1: WEBrick::HTTPVersion
         1: Fluent::EventRouter::MatchCache
         1: Fluent::EventRouter
         1: Fluent::NoMatchMatch
         1: Fluent::RootAgent
         1: Fluent::SystemConfig
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: IRB::Notifier::NoMsgNotifier
         1: Fluent::PluginLogger
         1: Fluent::Plugin::DebugAgentInput
         1: Fluent::Plugin::ForwardOutput
         1: Set
         1: OptionParser::CompletingHash
         1: Gem::Platform
         1: #<Class:0x007fb4ac4daf18>
         1: Gem::PathSupport
         1: #<Class:0x007fb4ac536b38>
         1: fatal
         1: Data
         1: Fluent::Plugin::FileBuffer
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: Complex
         1: IOError
         1: Binding
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
  String 3,842,341 bytes
   Array 1 elements
    Hash 223 pairs
GLStephen commented 6 years ago

note @repeatedly the deadlock posted above happens with flush_thread_count > 1 (I had it set to 8) almost immediately. With flush_thread_count set to 1 the deadlock certainly does not happen immediately, but I am waiting to see if it happens over a longer period of time. Hopefully that helps.

GLStephen commented 6 years ago

With threads at 1, it took about 2 hours to fail.

repeatedly commented 6 years ago

pid2line.rb result from slack

$ /opt/td-agent/embedded/bin/ruby pid2line.rb 887
ruby            state:S utime:99 LWPID:887 EIP:7fb4b456ea82 -> ba82
Elf vaddr: ba82 -> ba82 in /usr/lib64/libpthread-2.17.so
__pthread_cond_timedwait
:?
ruby-timer-thr  state:S utime:0 LWPID:888 EIP:7fb4b3b51dfd -> ecdfd
Elf vaddr: ecdfd -> ecdfd in /usr/lib64/libc-2.17.so
__poll_nocancel
:?
thread.rb:70    state:S utime:18 LWPID:991 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70    state:S utime:17 LWPID:992 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70    state:S utime:19 LWPID:993 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70    state:S utime:18 LWPID:994 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70    state:S utime:16 LWPID:995 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70    state:S utime:18 LWPID:996 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70    state:S utime:18 LWPID:997 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70    state:S utime:16 LWPID:998 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70    state:S utime:6 LWPID:999 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
thread.rb:70    state:S utime:60 LWPID:1000 EIP:7fb4b3b5cd13 -> f7d13
Elf vaddr: f7d13 -> f7d13 in /usr/lib64/libc-2.17.so
__epoll_wait_nocancel
:?
drb.rb:1498     state:S utime:0 LWPID:1001 EIP:7fb4b3b53ba3 -> eeba3
Elf vaddr: eeba3 -> eeba3 in /usr/lib64/libc-2.17.so
__select_nocancel
:?
thread.rb:70    state:S utime:6 LWPID:1002 EIP:7fb4b3b5cd13 -> f7d13
Elf vaddr: f7d13 -> f7d13 in /usr/lib64/libc-2.17.so
__epoll_wait_nocancel
:?
thread.rb:70    state:S utime:4 LWPID:1003 EIP:7fb4b3b53ba3 -> eeba3
Elf vaddr: eeba3 -> eeba3 in /usr/lib64/libc-2.17.so
__select_nocancel
:?
thread.rb:70    state:S utime:351 LWPID:1004 EIP:7fb4b456e6d5 -> b6d5
Elf vaddr: b6d5 -> b6d5 in /usr/lib64/libpthread-2.17.so
__pthread_cond_wait
:?
GLStephen commented 6 years ago

@repeatedly I've been working through this deadlock issue and I had a question about the retry process, is that a separate thread? or does it consume the thread that is active? How is retry stored? memory? is it reasonable that a huge number of chunks in a retry state could lead to what we are seeing here, or at least in my case? I was looking at some libraries and behavior and it seems that some of the plugins were stuck in a retry loop because of a library version that was upgraded from the gemspec that has an issue, the question is whether that matters, if that only consumes 1 thread and just keeps retrying the 1 chunk until it's tossed then it won't cause an increasing resource usage, if retries are piled up though the behavior would be different

repeatedly commented 6 years ago

I had a question about the retry process, is that a separate thread? or does it consume the thread that is active?

Does "retry process" mean buffer chunk retry, right? If so, retry is done by same thread for flush. Output plugin shares retry state and if the flush of one thread is succeeded, retry state is cleared.

How is retry stored? memory?

Yes. See the code: https://github.com/fluent/fluentd/blob/629c470a2a8412dd51875f8adbea0ea7b47dc914/lib/fluent/plugin/output.rb#L1134

it seems that some of the plugins were stuck in a retry loop because of a library version that was upgraded from the gemspec that has an issue,

Does this 'stuck' is different from this issue? Is it known issue between plugin/library versions?

it won't cause an increasing resource usage,

Yeah. retry itself doesn't much increase resource usage.

Does this deadlock happen after some errors?

abicky commented 6 years ago

I tried to reproduce this issue in 629c470a2a8412dd51875f8adbea0ea7b47dc914 by the following steps and I found that deadlock occurs if an input plugin thread locks chunks in Buffer#write by chunk.mon_enter and an enqueue thread locks the buffer in Buffer#enqueue_chunk at the same time.

1. Add the following change

diff --git a/lib/fluent/plugin/buffer.rb b/lib/fluent/plugin/buffer.rb
index be66f3e8..d4c6be87 100644
--- a/lib/fluent/plugin/buffer.rb
+++ b/lib/fluent/plugin/buffer.rb
@@ -496,6 +496,7 @@ module Fluent

           if stored
             block.call(chunk, adding_bytesize)
+            sleep 3
           end
         end

2. Prepare a configuration file

<source>
  @type forward
</source>

<match test.*>
  @type file
  format json
  path /tmp/fluentd/file/${tag}-%Y%m%d

  <buffer time, tag>
    path /tmp/fluentd/buffer/test.*.buffer
    flush_mode interval
    flush_interval 1s
    timekey 1d
  </buffer>
</match>

3. Launch fluentd

ruby bin/fluentd -c deadlock.conf

4. Post events

require 'fluent-logger'
logger = Fluent::Logger::FluentLogger.new(nil, host: 'localhost', port: 24224)
logger.send(:write, ['test.a', [[Time.now.to_i - 86400, { msg: 0 }], [Time.now.to_i, { msg: 1 }]]])

5. sigdump

Sigdump at 2017-10-11 17:24:33 +0900 process 84647 (bin/fluentd)
  Thread #<Thread:0x007fb08107ef70> status=run priority=0
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `backtrace'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:52:in `dump_backtrace'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:34:in `block in dump_all_thread_backtrace'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `each'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:33:in `dump_all_thread_backtrace'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:16:in `block in dump'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `open'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:136:in `_open_dump_path'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/sigdump-0.2.4/lib/sigdump.rb:7:in `block in setup'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/engine.rb:228:in `sleep'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/engine.rb:228:in `run'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/supervisor.rb:774:in `run_engine'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/supervisor.rb:523:in `block in run_worker'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/supervisor.rb:699:in `main_process'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/supervisor.rb:518:in `run_worker'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
      bin/fluentd:5:in `<main>'
  Thread #<Thread:0x007fb080c6e6b0> status=sleep priority=0
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:304:in `queued?'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:1015:in `next_flush_time'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:1289:in `flush_thread_run'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:438:in `block (2 levels) in start'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb080c6e430> status=sleep priority=0
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:320:in `block in enqueue_chunk'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:316:in `enqueue_chunk'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:356:in `block (2 levels) in enqueue_all'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:353:in `each'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:353:in `block in enqueue_all'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:351:in `enqueue_all'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:1249:in `enqueue_thread_run'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  Thread #<Thread:0x007fb080c6e228> status=sleep priority=0
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `lock'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:187:in `mon_enter'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/2.4.0/monitor.rb:212:in `mon_synchronize'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:457:in `write_once'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:207:in `block in write'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:206:in `each'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/buffer.rb:206:in `write'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:897:in `block in handle_stream_with_custom_format'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:823:in `write_guard'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:896:in `handle_stream_with_custom_format'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:815:in `execute_chunking'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/output.rb:750:in `emit_buffered'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/event_router.rb:96:in `emit_stream'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:320:in `on_message'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:211:in `block in handle_connection'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:248:in `block (3 levels) in read_messages'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:247:in `feed_each'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:247:in `block (2 levels) in read_messages'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin/in_forward.rb:256:in `block in read_messages'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin_helper/server.rb:576:in `on_read_without_connection'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/cool.io-1.5.1/lib/cool.io/io.rb:123:in `on_readable'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/cool.io-1.5.1/lib/cool.io/io.rb:186:in `on_readable'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/cool.io-1.5.1/lib/cool.io/loop.rb:88:in `run_once'
      /Users/arabiki/.anyenv/envs/rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/cool.io-1.5.1/lib/cool.io/loop.rb:88:in `run'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin_helper/event_loop.rb:84:in `block in start'
      /Users/arabiki/.ghq/github.com/fluent/fluentd/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  GC stat:
      count: 54
      heap_allocated_pages: 377
      heap_sorted_length: 409
      heap_allocatable_pages: 32
      heap_available_slots: 153663
      heap_live_slots: 153127
      heap_free_slots: 536
      heap_final_slots: 0
      heap_marked_slots: 100124
      heap_eden_pages: 377
      heap_tomb_pages: 0
      total_allocated_pages: 377
      total_freed_pages: 0
      total_allocated_objects: 1135049
      total_freed_objects: 981922
      malloc_increase_bytes: 5486912
      malloc_increase_bytes_limit: 16777216
      minor_gc_count: 47
      major_gc_count: 7
      remembered_wb_unprotected_objects: 1162
      remembered_wb_unprotected_objects_limit: 2320
      old_objects: 97483
      old_objects_limit: 194974
      oldmalloc_increase_bytes: 5710432
      oldmalloc_increase_bytes_limit: 16777216
  Built-in objects:
   153,663: TOTAL
    76,920: T_STRING
    34,229: T_ARRAY
    17,853: T_IMEMO
    11,396: T_OBJECT
     5,465: T_NODE
     2,324: T_CLASS
     2,125: T_HASH
     1,580: T_DATA
       543: T_REGEXP
       460: FREE
       255: T_ICLASS
       232: T_MODULE
        71: T_RATIONAL
        66: T_STRUCT
        58: T_SYMBOL
        28: T_MATCH
        26: T_BIGNUM
        22: T_FILE
         9: T_FLOAT
         1: T_COMPLEX
  All objects:
    69,320: String
    25,693: Array
     4,931: Gem::Requirement
     3,377: Gem::Dependency
     1,899: Hash
     1,318: Class
       844: Time
       769: Gem::StubSpecification
       769: Gem::StubSpecification::StubLine
       769: Gem::Specification
       596: Gem::Version
       543: Regexp
       382: Proc
       232: Module
       101: Encoding
        76: RubyVM::InstructionSequence
        71: Rational
        69: Fluent::Config::ConfigureProxy
        58: Symbol
        53: Range
        46: MatchData
        30: Thread::Mutex
        28: Binding
        26: Integer
        17: OptionParser::Switch::RequiredArgument
        16: File
        14: Thread::Backtrace
        13: OptionParser::Switch::NoArgument
        10: Fluent::Config::Section
         9: Float
         8: Method
         8: Fluent::Config::Element
         8: Fluent::Registry
         6: OptionParser::OptionMap
         4: Thread
         4: Coolio::IO::Watcher
         4: Fluent::Plugin::Buffer::Metadata
         4: Fluent::Plugin::Base::State
         4: Object
         3: IO
         3: OptionParser::List
         3: JSON::Ext::Generator::State
         3: IRB::Notifier::LeveledNotifier
         3: UnboundMethod
         3: MessagePack::Buffer
         2: Fluent::PluginHelper::Server::EventHandler::TCPServer
         2: OptionParser::Switch::PlacedArgument
         2: OptionParser::Switch::OptionalArgument
         2: ThreadSafe::Cache
         2: BigDecimal
         2: Fluent::PluginHelper::Server::ServerInfo
         2: IO::Buffer
         2: OpenSSL::PKey::DH
         2: Monitor
         2: MessagePack::Unpacker
         2: Fluent::Plugin::Output::PlaceholderValidator
         2: MessagePack::Factory
         2: Fluent::Plugin::Buffer::FileChunk
         1: TCPSocket
         1: StringScanner
         1: Fluent::Log
         1: Logger::Formatter
         1: ServerEngine::DaemonLogger
         1: WEBrick::HTTPVersion
         1: #<Class:0x007fb08082e208>
         1: #<Class:0x007fb0808547c8>
         1: IRB::Notifier::CompositeNotifier
         1: IRB::StdioOutputMethod
         1: IRB::Notifier::NoMsgNotifier
         1: Errno::EEXIST
         1: Fluent::Compat::NullOutputChain
         1: Fluent::Plugin::ForwardInput
         1: LoadError
         1: Fluent::Plugin::JSONFormatter
         1: ArgumentError
         1: Coolio::Loop
         1: TCPServer
         1: Fluent::PluginHelper::EventLoop::DefaultWatcher
         1: Fluent::Plugin::Output::FlushThreadState
         1: Complex
         1: ThreadGroup
         1: IOError
         1: RubyVM
         1: NoMemoryError
         1: SystemStackError
         1: Random
         1: ARGF.class
         1: fatal
         1: Data
         1: Gem::PathSupport
         1: Set
         1: OptionParser::CompletingHash
         1: Gem::Platform
         1: Fluent::EngineClass
         1: URI::RFC2396_Parser
         1: URI::RFC3986_Parser
         1: OpenSSL::X509::Store
         1: Resolv::DNS::Config
         1: Resolv::DNS
         1: Resolv::Hosts
         1: Resolv
         1: Fluent::EventRouter::MatchCache
         1: Fluent::EventRouter
         1: Fluent::NoMatchMatch
         1: Fluent::RootAgent
         1: Fluent::SystemConfig
         1: Fluent::Supervisor::LoggerInitializer
         1: Fluent::Supervisor
         1: OptionParser
         1: MessagePack::Packer
         1: Fluent::PluginHelper::Server::EventHandler::UDPServer
         1: UDPSocket
         1: IPAddr
         1: Coolio::TCPServer
         1: Fluent::Plugin::FileOutput
         1: Fluent::Plugin::FileBuffer
         1: Fluent::GlobMatchPattern
         1: Fluent::EventRouter::Rule
         1: Fluent::EventTime
         1: Fluent::MultiEventStream
         1: Fluent::PluginHelper::Server::TCPCallbackSocket
  String 4,318,695 bytes
   Array 3 elements
    Hash 0 pairs
repeatedly commented 6 years ago

@abicky Thanks. I will check it!

repeatedly commented 6 years ago

It is reproduced on my env. The problem is no buffer lock in write. It causes lock order mismatch. For the performance, giant lock in write is not good. I'm now trying to fix the problem.

GLStephen commented 6 years ago

@repeatedly I'm not 100% up on fluentd internals so just trying to be helpful, but is it possible to create a system like used for timekey and timekey_wait, but in reverse. Stop writing to a buffer chunk before it is enqueud, then when enqueue occurs the lock is simply unnecessary. This could operate by monitoring "flush_percentage" or something similar. If the lock is problematic, try to avoid needing one.

abicky commented 6 years ago

@repeatedly Thank you for your effort :) Do you have any update on this issue? If you are busy to fix the problem, I'll also try to fix it.

repeatedly commented 6 years ago

@abicky If you have a patch for this, it is very helpful. Because we don't have actual production environment for this problem, so if your patch resolve your production issue, this is reliable for other users :)

abicky commented 6 years ago

@repeatedly Okay, I'll try to write a patch 💪

mkishi commented 6 years ago

@abicky Currently, I am facing the same issue. I hope this patch, too !

mpeltonen commented 6 years ago

@repeatedly @abicky We are seeing this issue happening all the time (v14.22.rc1). Our archive servers deadlock in about 5 minutes from restart. Using append and compress may make this happen more easily(?).

Here's our out_file configuration:

<match **>
  @type file
  path /data/log/${tag}/${tag}
  <buffer tag,time>
    @type file
    path /data/buffer
    flush_mode interval
    flush_interval 10
    flush_at_shutdown true
    total_limit_size 1G
    timekey_use_utc true
  </buffer>
  <format>
    @type json
  </format>
  include_time_key true
  append true
  compress gzip
</match>
abicky commented 6 years ago

I've created https://github.com/fluent/fluentd/pull/1721 to resolve this issue.

repeatedly commented 6 years ago

Does anyone test above patch on your traffic / setting?

abicky commented 6 years ago

As I commented in https://github.com/fluent/fluentd/pull/1721#issuecomment-338371066, I've checked it worked fine in our development environment although the number of events is less than the number in the production environment. I'll try it in our production environment after the release of a new RC version or your review. In our production environment deadlock occurs almost everyday, so it takes a few days to test it in a more complicated environment.

Of course, I checked the patch resolved the problem in the case of https://github.com/fluent/fluentd/issues/1549#issuecomment-335736547.

dasterintempus commented 6 years ago

I've just applied the patched gem and have started testing it. Usually my environment was deadlocking in 1-3 hours, I should be able to report back tomorrow with the results.

dasterintempus commented 6 years ago

so far so good! 22 hours in and no deadlocks, running with 4 worker processes. @repeatedly any ideas when this can hit an RC or actual release version? Big thank you to @abicky for the patch!

GLStephen commented 6 years ago

@scottbuettner any notable impact on performance? I'm very encouraged that this patch will solve the issues we've been seeing also!

dasterintempus commented 6 years ago

@GLStephen I don't have any benchmarks to compare against or anything that detailed, but I didn't notice a drop in performance from just comparing CPU/load/etc.

repeatedly commented 6 years ago

I released v0.14.22.rc2 for testing.

mpeltonen commented 6 years ago

Looks promising for us too! No deadlocks seen so far with .22.rc2. Thanks a lot @abicky for the patch!

GLStephen commented 6 years ago

thanks for the patch @bicky , @repeatedly this seems to fix our issue as well, we'll fully load it over the weekend and know for sure, but we are not seeing the problematic behaviors we were seeing before at all