search

fluent / fluent-plugin-mongo

MongoDB input and output plugin for Fluentd
https://docs.fluentd.org/output/mongo
173 stars 61 forks source link

error_class="Mongo::ConnectionTimeoutError" #59

Closed crazw closed 9 years ago

crazw commented 9 years ago

Hi, my conf file is like:

<match fx*>
  # plugin type
  type mongo

  # mongodb db + collection
  database bigdata
  collection log_store

  # mongodb host + port
  host 127.0.0.1
  port 27017

  # interval
  flush_interval 10s

  # buffer_queue_limit
  buffer_queue_limit 256mb
  buffer_chunk_limit 8388608
  num_threads 3

  # make sure to include the time key
  include_time_key true
</match>

And when I check the logs, It give me a error:

2015-08-31 09:16:33 +0800 [warn]: temporarily failed to flush the buffer. next_retry=2015-08-31 09:16:29 +0800 error_class="Mongo::ConnectionTimeoutError" error="could not obtain connection within 5.0 seconds. The max pool size is currently 1; consider increasing the pool size or timeout." plugin_id="object:3f9ca6a16080"
  2015-08-31 09:16:33 +0800 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.2/lib/mongo/connection/pool.rb:261:in `block in checkout'
  2015-08-31 09:16:33 +0800 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.2/lib/mongo/connection/pool.rb:259:in `loop'
  2015-08-31 09:16:33 +0800 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/mongo-1.12.2/lib/mongo/connection/pool.rb:259:in `checkout'

Can you give me some suggestions?? Thank you very much!

repeatedly commented 9 years ago

Currenlty nothing. Maybe your mongodb is slow or lack of capacity.

crazw commented 9 years ago

Is there a config field that increasing the pool size or timeout

repeatedly commented 9 years ago

Nothing for now. Hmm... with num_threads, specifying max_pool_size parameter may relax this problem.

crazw commented 9 years ago

How can I specify max_pool_size parameter？

# buffer_queue_limit
  buffer_queue_limit 256mb
  buffer_chunk_limit 8388608
  num_threads 20
  max_pool_size 20

  # log_level option
  log_level fatal

it show me:

[warn]: parameter 'max_pool_size' in xxxxxxxxxxxxxx is not used.

Thank you very much.

repeatedly commented 9 years ago

Currently no support. Patches are welcome.

crazw commented 9 years ago

this is my mongodb activity monitor: image

there is still some "0" about insert/s . my fluentd's conf is : 

<match fxdata>
  # plugin type
  type mongo

  # mongodb db + collection
  database bigdata
  collection log_store

  # mongodb host + port
  host 192.168.1.223
  port 27017

  # interval
  flush_interval 1s

  # buffer_queue_limit
  buffer_queue_limit 256mb
  buffer_chunk_limit 8388608
  num_threads 20
#  max_pool_size 20

  # log_level option
  log_level fatal

  # make sure to include the time key
#  include_time_key true
</match>

Can you tell how can I optimize it. Thanks.

crazw commented 9 years ago

Can you give me some suggests? How can I change the pool size or timeout in source code?

sethest commented 9 years ago

I had the same problem as you ! After trial and error, I found num_threads cause this problem. I got warning when setting num_threads 2 # or above , and there were some duplicate events in my destination(MongoDB). Please mark parameter num_threads (default = 1) , then there is no warning anymore. I wish you understand my poor English~

repeatedly commented 9 years ago

@crazw

How can I change the pool size or timeout in source code?

I'm not sure. Maybe mongo ruby client should have these parameters. I don't have good test environment for now so I want to check it on your environment.

@sethest

and there were some duplicate events in my destination(MongoDB).

In num_threads, fluentd sends different chunks in one thread so it should not be duplicated. Maybe mongo ruby client has wrong mechanizm.

BTW, could you test above pool size parameter with num_threads?

sethest commented 9 years ago

I am not good at programming in any language, even modify plugin or library. So I don't understand what to do , sorry.

repeatedly commented 9 years ago

I wrote a patch for this issue > https://github.com/fluent/fluent-plugin-mongo/pull/62

Does someone test this patch?