search

fluent / fluent-plugin-mongo

MongoDB input and output plugin for Fluentd
https://docs.fluentd.org/output/mongo
173 stars 61 forks source link

Collection created is not capped #72

Closed Nepoxx closed 8 years ago

Nepoxx commented 8 years ago

This is the fluentd config I have:

<match mongo.**>
  type mongo

  database fluentd

  # tags are collections
  tag_mapped

  capped
  capped_size 1024m

  remove_tag_prefix mongo.

  host 127.0.0.1
  port 27017

  flush_interval 10s

  include_time_key true
</match>

In mongo shell:

> db.dev.apache.access.isCapped()
false
> db.version()
3.2.4

It would seem the db is not capped as it should. I tried dropping the database and restarting fluentd, same result.

repeatedly commented 8 years ago

Weird. On my environmen, it works. If other user creates a table before fleuntd access, fluentd keeps previous configuration in tag_mapped mode. Could you re-check it?

% ./bin/mongo
MongoDB shell version: 3.2.4
connecting to: test
Server has startup warnings: 
2016-03-16T01:26:19.911+0900 I CONTROL  [initandlisten] 
2016-03-16T01:26:19.911+0900 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
> show dbs;
fluentd  0.000GB
local    0.000GB
> use fluentd;
switched to db fluentd
> show tables;
test
> db.version();
3.2.4
> db.test.isCapped();
true
> 
2016-03-16 01:27:31 +0900 [info]: reading config file path="mongo.conf"
2016-03-16 01:27:31 +0900 [info]: starting fluentd-0.12.21
# ...
2016-03-16 01:27:31 +0900 [info]: gem 'fluent-plugin-mongo' version '0.7.12'
# ...
2016-03-16 01:27:31 +0900 [info]: using configuration file: <ROOT>
  <source>
    @type forward
  </source>
  <match mongo.**>
    type mongo
    database fluentd
    tag_mapped 
    capped 
    capped_size 1024m
    remove_tag_prefix mongo.
    host 127.0.0.1
    port 27017
    flush_interval 10s
    include_time_key true
  </match>
</ROOT>
$ echo '{"k":"v"}' | ~/dev/fluentd/fluentd/bin/fluent-cat mongo.test
Nepoxx commented 8 years ago 
sudo /etc/init.d/td-agent stop
Stopping td-agent:  * td-agent

> show dbs
fluentd        0.000GB
local          0.000GB
> use fluentd
switched to db fluentd
> db.dropDatabase()
{ "dropped" : "fluentd", "ok" : 1 }
> show dbs
local          0.000GB

sudo /etc/init.d/td-agent start
Starting td-agent:  * td-agent

fluent log:

2016-03-15 13:13:04 -0400 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-03-15 13:13:04 -0400 [info]: starting fluentd-0.12.20
2016-03-15 13:13:04 -0400 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.1'
2016-03-15 13:13:04 -0400 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-03-15 13:13:04 -0400 [info]: gem 'fluent-plugin-mongo' version '0.7.12'
2016-03-15 13:13:04 -0400 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'
2016-03-15 13:13:04 -0400 [info]: gem 'fluent-plugin-s3' version '0.6.5'
2016-03-15 13:13:04 -0400 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-03-15 13:13:04 -0400 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-03-15 13:13:04 -0400 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-03-15 13:13:04 -0400 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2016-03-15 13:13:04 -0400 [info]: gem 'fluentd' version '0.12.20'
2016-03-15 13:13:04 -0400 [info]: adding match pattern="debug.**" type="stdout"
2016-03-15 13:13:04 -0400 [info]: adding match pattern="mongo.**" type="mongo"
2016-03-15 13:13:04 -0400 [info]: adding source type="debug_agent"
2016-03-15 13:13:04 -0400 [info]: adding source type="tail"
2016-03-15 13:13:04 -0400 [info]: using configuration file: <ROOT>
  <match debug.**>
    type stdout
  </match>
  <match mongo.**>
    type mongo
    database fluentd
    tag_mapped 
    capped 
    capped_size 1024m
    remove_tag_prefix mongo.
    host 127.0.0.1
    port 27017
    flush_interval 10s
    include_time_key true
  </match>
  <source>
    type debug_agent
    bind 127.0.0.1
    port 24230
  </source>
  <source>
    type tail
    format json
    path /var/log/apache2/dev.sarbakan.com/access.log
    pos_file /var/log/td-agent/dev.sarbakan.com.access.log.pos
    tag mongo.dev.apache.access
    time_key time
    time_format %FT%T%:z
  </source>
</ROOT>
2016-03-15 13:13:04 -0400 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
2016-03-15 13:13:04 -0400 [info]: following tail of /var/log/apache2/dev.sarbakan.com/access.log

and then

> show dbs
fluentd        0.000GB
local          0.000GB
> use fluentd
switched to db fluentd
> db.dev.apache.access.isCapped()
false
Nepoxx commented 8 years ago

This is probably an issue with my mongo shell, I'll open the issue with them. I tried the following which implies that the collection is capped even though my shell reports it as not being capped

> db.dev.apache.access.remove({})
WriteResult({
    "nRemoved" : 0,
    "writeError" : {
        "code" : 20,
        "errmsg" : "cannot remove from a capped collection: fluentd.dev.apache.access"
    }
})
> db.dev.apache.access.isCapped()
false

Sorry!