Enable Dynamic Thread Pool to Improve Memory Usage

[qingling128]

During our benchmark tests, we found that the memory usage of Fluentd increases as num_threads increases (we keep the other configurations exactly the same). It's almost linear.

We are testing with pretty low QPS, which should not require that many threads as specified in num_threads. It makes sense to pre-allocate memory for each thread. But seems Fluentd could have saved a lot of resource if it uses some dynamic thread pool / kill idle threads and add new ones only when necessary. That way the baseline of memory usage should be lower.

Is this something that's possible to enable with the current Fluentd (we haven't tried v1.0.X yet)?

Thanks!

• Fluentd version v0.14.25
• OS ubuntu-1604-xenial
• Configuration

  
  <source>
  @type forward
  port 24224
  bind 127.0.0.1
  </source>

@type prometheus port 24231

@type prometheus_monitor

<match **> @type google_cloud buffer_chunk_limit 1M flush_interval 5s max_retry_wait 300 disable_retry_limit num_threads 8 use_grpc true enable_monitoring true monitoring_type prometheus

[repeatedly]

Idea looks good. We may use Thread Pool in concurrent-ruby for such purpose: https://github.com/ruby-concurrency/concurrent-ruby/blob/master/doc/thread_pools.md But we need time to investigate how to adapt this model in output core.

[qingling128]

Thanks @repeatedly ! For now we will suggest customers to increase num_threads with caution and be aware of the memory usage implication.