MongoDB and Ruby-Driver have a size limit of insert operation.

[repeatedly]

Fluentd has a buffer, so Fluentd stores some objects to its buffer. MongoDB or Ruby-Driver raise "Exceded maximum insert size of 16,000,000 bytes" exception when buffering size gets over the size limit of insert operation.

We can check MongoDB version and reset Fluentd's buffer size to version-dependent MongoDB's max size.

[frsyuki]

Consider to apply following patch:

diff --git a/lib/fluent/plugin/out_mongo.rb b/lib/fluent/plugin/out_mongo.rb
index 58425c3..c986803 100644
--- a/lib/fluent/plugin/out_mongo.rb
+++ b/lib/fluent/plugin/out_mongo.rb
@@ -26,6 +26,9 @@ class MongoOutput < BufferedOutput
     @argument = {:capped => false}
   end

+  MAX_BUFFER_SIZE_IN_BSON_FORMAT = 16_000_000  # TODO depends on mongo version
+  MAX_BUFFER_SIZE_IN_MSGPACK_FORMAT = MAX_BUFFER_SIZE_IN_BSON_FORMAT*0.7  # TODO depends on data
+
   def configure(conf)
     super

@@ -41,6 +44,12 @@ class MongoOutput < BufferedOutput
     def @timef.format_nocache(time)
       time
     end
+
+    if @buffer.respond_to?(:buffer_chunk_limit) &&
+        @buffer.buffer_chunk_limit > MAX_BUFFER_SIZE_IN_MSGPACK_FORMAT
+      @buffer.buffer_chunk_limit = MAX_BUFFER_SIZE_IN_MSGPACK_FORMAT
+      # TODO warning?
+    end
   end

   def start