search

fluent / fluent-plugin-mongo

MongoDB input and output plugin for Fluentd
https://docs.fluentd.org/output/mongo
173 stars 61 forks source link

Fluent keeps way too few records #39

Closed ghost closed 9 years ago

ghost commented 9 years ago

Here's my conf:

<source>
  type tail
  path /var/lib/docker/containers/*/*-json.log
  pos_file /var/log/fluentd-docker.pos
  time_format %Y-%m-%dT%H:%M:%S
  tag docker.*
  format json
</source>

<match docker.var.lib.docker.containers.*.*.log>
  type record_reformer
  container_id ${tag_parts[5]}
  tag docker.all
</match>

<match docker.all>

  # plugin type
  type mongo

  # mongodb db + collection
  database main
  collection fluents

  # for capped collection
  capped
  capped_size 5120m

  # json
  format json

  # interval
  flush_interval 2s

  # make sure to include the time key
  include_time_key true

  host xxx
  port xxx

</match>

Running two consecutive queries:

> db.fluents.find()
{ "_id" : ObjectId("54a2df06f058ac000f001488"), "log" : "154.122.124.96 - - [30/Dec/2014:17:21:10 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36\" \"-\"\n", "stream" : "stdout", "container_id" : "e0b87ff04c2999d0a70e1d623b0f1f0293d3430486544d7319e7fa667555a268", "time" : ISODate("2014-12-30T17:21:10Z") }
{ "_id" : ObjectId("54a2df08f058ac00260000b5"), "log" : "154.122.124.96 - - [30/Dec/2014:17:21:10 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36\" \"-\"\n", "stream" : "stdout", "container_id" : "e0b87ff04c2999d0a70e1d623b0f1f0293d3430486544d7319e7fa667555a268", "time" : ISODate("2014-12-30T17:21:10Z") }
{ "_id" : ObjectId("54a2df08f058ac00260000b6"), "log" : "2014/12/30 17:21:11 [error] 7#0: *30 open() \"/usr/share/nginx/html/favicon.ico\" failed (2: No such file or directory), client: 154.122.124.96, server: localhost, request: \"GET /favicon.ico HTTP/1.1\", host: \"31.3.0.19:43306\"\n", "stream" : "stderr", "container_id" : "e0b87ff04c2999d0a70e1d623b0f1f0293d3430486544d7319e7fa667555a268", "time" : ISODate("2014-12-30T17:21:11Z") }
{ "_id" : ObjectId("54a2df08f058ac00260000b7"), "log" : "154.122.124.96 - - [30/Dec/2014:17:21:11 +0000] \"GET /favicon.ico HTTP/1.1\" 404 570 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36\" \"-\"\n", "stream" : "stdout", "container_id" : "e0b87ff04c2999d0a70e1d623b0f1f0293d3430486544d7319e7fa667555a268", "time" : ISODate("2014-12-30T17:21:11Z") }
{ "_id" : ObjectId("54a2df09f058ac000f001489"), "log" : "2014/12/30 17:21:11 [error] 7#0: *30 open() \"/usr/share/nginx/html/favicon.ico\" failed (2: No such file or directory), client: 154.122.124.96, server: localhost, request: \"GET /favicon.ico HTTP/1.1\", host: \"31.3.0.19:43306\"\n", "stream" : "stderr", "container_id" : "e0b87ff04c2999d0a70e1d623b0f1f0293d3430486544d7319e7fa667555a268", "time" : ISODate("2014-12-30T17:21:11Z") }
{ "_id" : ObjectId("54a2df09f058ac000f00148a"), "log" : "154.122.124.96 - - [30/Dec/2014:17:21:11 +0000] \"GET /favicon.ico HTTP/1.1\" 404 570 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36\" \"-\"\n", "stream" : "stdout", "container_id" : "e0b87ff04c2999d0a70e1d623b0f1f0293d3430486544d7319e7fa667555a268", "time" : ISODate("2014-12-30T17:21:11Z") }
{ "_id" : ObjectId("54a2df4bf058ac000f00148b"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon] mem (MB) res:65 virt:543\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2df4bf058ac000f00148c"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon]  mapped (incl journal view):320\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2df4bf058ac000f00148d"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon]  connections:25\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2df4df058ac00260000b8"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon] mem (MB) res:65 virt:543\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2df4df058ac00260000b9"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon]  mapped (incl journal view):320\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2df4df058ac00260000ba"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon]  connections:25\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
> db.fluents.find()
{ "_id" : ObjectId("54a2df09f058ac000f001489"), "log" : "2014/12/30 17:21:11 [error] 7#0: *30 open() \"/usr/share/nginx/html/favicon.ico\" failed (2: No such file or directory), client: 154.122.124.96, server: localhost, request: \"GET /favicon.ico HTTP/1.1\", host: \"31.3.0.19:43306\"\n", "stream" : "stderr", "container_id" : "e0b87ff04c2999d0a70e1d623b0f1f0293d3430486544d7319e7fa667555a268", "time" : ISODate("2014-12-30T17:21:11Z") }
{ "_id" : ObjectId("54a2df09f058ac000f00148a"), "log" : "154.122.124.96 - - [30/Dec/2014:17:21:11 +0000] \"GET /favicon.ico HTTP/1.1\" 404 570 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36\" \"-\"\n", "stream" : "stdout", "container_id" : "e0b87ff04c2999d0a70e1d623b0f1f0293d3430486544d7319e7fa667555a268", "time" : ISODate("2014-12-30T17:21:11Z") }
{ "_id" : ObjectId("54a2df4bf058ac000f00148b"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon] mem (MB) res:65 virt:543\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2df4bf058ac000f00148c"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon]  mapped (incl journal view):320\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2df4bf058ac000f00148d"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon]  connections:25\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2df4df058ac00260000b8"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon] mem (MB) res:65 virt:543\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2df4df058ac00260000b9"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon]  mapped (incl journal view):320\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2df4df058ac00260000ba"), "log" : "2014-12-30T17:22:19.113+0000 [clientcursormon]  connections:25\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:22:19Z") }
{ "_id" : ObjectId("54a2e077f058ac000f00148e"), "log" : "2014-12-30T17:27:19.169+0000 [clientcursormon] mem (MB) res:65 virt:543\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:27:19Z") }
{ "_id" : ObjectId("54a2e077f058ac000f00148f"), "log" : "2014-12-30T17:27:19.169+0000 [clientcursormon]  mapped (incl journal view):320\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:27:19Z") }
{ "_id" : ObjectId("54a2e077f058ac000f001490"), "log" : "2014-12-30T17:27:19.169+0000 [clientcursormon]  connections:25\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:27:19Z") }
{ "_id" : ObjectId("54a2e079f058ac00260000bb"), "log" : "2014-12-30T17:27:19.169+0000 [clientcursormon] mem (MB) res:65 virt:543\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:27:19Z") }
{ "_id" : ObjectId("54a2e079f058ac00260000bc"), "log" : "2014-12-30T17:27:19.169+0000 [clientcursormon]  mapped (incl journal view):320\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:27:19Z") }
{ "_id" : ObjectId("54a2e079f058ac00260000bd"), "log" : "2014-12-30T17:27:19.169+0000 [clientcursormon]  connections:25\n", "stream" : "stdout", "container_id" : "3fba10c1a8e54fdb66e69d6c03147fcf40f9871bede827bc42e0eef0918c9634", "time" : ISODate("2014-12-30T17:27:19Z") }

Any idea why? since the disk size is not an issue. Thanks in advance.

repeatedly commented 9 years ago

Sorry, I don't understand what you say. What does "keeps way too few records" mean? Does db.fluents.find().size() return always same or smaller result? If so, you should check your collection settings.

ghost commented 9 years ago

Had to recreate the collection .. works perfectly. Shouldn't the plugin apply new cap size? Anyway, great work!

repeatedly commented 9 years ago

Had to recreate the collection .. works perfectly.

Good :+1:

Shouldn't the plugin apply new cap size?

fluentd can't judge existing collection is correct or not. So if target collect exists, fluentd keeps such collection.