Working with DynamoDB Local

[jgfrancisco]

I'm trying to evaluate fluentd with dynamodb. I am trying to set-up fluentd and dynamodb-local both running in docker containers. I'm having problems with making fluentd communicate.

My fluent.conf reads:

<source>
  type tail
  path /var/lib/docker/containers/*/*-json.log
  pos_file /var/log/fluentd-docker.pos
  time_format %Y-%m-%dT%H:%M:%S
  tag docker.*
  format json
</source>

<match docker.var.lib.docker.containers.*.*.log>
  type record_reformer
  container_id ${tag_parts[5]}
  tag docker.all
</match>

<match docker.all>
  type copy
  <store>
    type file
    path /var/log/docker/*.log
    format json
  </store>
  <store>
    type dynamodb
    aws_key_id test
    aws_sec_key test
    dynamo_db_endpoint http://172.17.8.101:8000
    dynamo_db_table fluentd_logs
  </store>
</match>

I'm getting the following errors:

2014-12-03 09:08:36 +0000 [info]: adding source type="tail"
2014-12-03 09:08:36 +0000 [info]: adding match pattern="docker.var.lib.docker.containers.*.*.log" type="record_reformer"
2014-12-03 09:08:36 +0000 [info]: adding match pattern="docker.all" type="dynamodb"
2014-12-03 09:08:36 +0000 [info]: detached process class=Fluent::DynamoDBOutput pid=14
2014-12-03 09:08:36 +0000 [fatal]: UnknownError: 'uninitialized constant Fluent::DynamoDBOutput::AWS'
2014-12-03 09:08:36 +0000 [info]: detached process class=Fluent::DynamoDBOutput pid=20
2014-12-03 09:08:36 +0000 [info]: following tail of /var/lib/docker/containers/1511a8b788cdb935e0486fbe63f64498e55dd398301f14a4f8c8b77ba598fd13/1511a8b788cdb935e0486fbe63f64498e55dd398301f14a4f8c8b77ba598fd13-json.log
2014-12-03 09:08:36 +0000 [info]: following tail of /var/lib/docker/containers/0147b85b8ebb8e55aad0f5640e0953d2b703493be11fce28756ce752f91364e7/0147b85b8ebb8e55aad0f5640e0953d2b703493be11fce28756ce752f91364e7-json.log
2014-12-03 09:08:36 +0000 [info]: following tail of /var/lib/docker/containers/526577b0a38d4c88a7c778242128578cb45f4488c663078bee66cf95138d82d4/526577b0a38d4c88a7c778242128578cb45f4488c663078bee66cf95138d82d4-json.log
2014-12-03 09:08:36 +0000 [info]: following tail of /var/lib/docker/containers/7aba28abc46370339c91ea1207fa8445d3c6827b54285c433786f5b5c5d0b02f/7aba28abc46370339c91ea1207fa8445d3c6827b54285c433786f5b5c5d0b02f-json.log
2014-12-03 09:08:36 +0000 [info]: following tail of /var/lib/docker/containers/a3cadf132509f0f928a23a57c45b744b1e658e9396b365f3a6e645ef6d8cf141/a3cadf132509f0f928a23a57c45b744b1e658e9396b365f3a6e645ef6d8cf141-json.log
2014-12-03 09:08:36 +0000 [info]: following tail of /var/lib/docker/containers/902719735ae7fabbf7bff058224ca354c4a01c10240d0142dea8680bded6d295/902719735ae7fabbf7bff058224ca354c4a01c10240d0142dea8680bded6d295-json.log
2014-12-03 09:08:36 +0000 [info]: following tail of /var/lib/docker/containers/80432f126376f914fecaf33c11d6b56d0ba8278f276417992416ac5e0572e085/80432f126376f914fecaf33c11d6b56d0ba8278f276417992416ac5e0572e085-json.log
2014-12-03 09:08:36 +0000 [fatal]: UnknownError: 'uninitialized constant Fluent::DynamoDBOutput::AWS'
2014-12-03 09:08:36 +0000 [info]: following tail of /var/lib/docker/containers/5df5977f3462061a4cf7b3c6f32cb65423550f8db8d60333202935c64034eb04/5df5977f3462061a4cf7b3c6f32cb65423550f8db8d60333202935c64034eb04-json.log
...
2014-12-03 09:08:36 +0000 [error]: error on forwerder thread error="Broken pipe"
  2014-12-03 09:08:36 +0000 [error]: /usr/local/lib/ruby/gems/2.1.0/gems/fluentd-0.10.56/lib/fluent/process.rb:253:in `write'
  2014-12-03 09:08:36 +0000 [error]: /usr/local/lib/ruby/gems/2.1.0/gems/fluentd-0.10.56/lib/fluent/process.rb:253:in `to_msgpack'
  2014-12-03 09:08:36 +0000 [error]: /usr/local/lib/ruby/gems/2.1.0/gems/fluentd-0.10.56/lib/fluent/process.rb:253:in `block in run'
  2014-12-03 09:08:36 +0000 [error]: /usr/local/lib/ruby/gems/2.1.0/gems/fluentd-0.10.56/lib/fluent/process.rb:251:in `each'
  2014-12-03 09:08:36 +0000 [error]: /usr/local/lib/ruby/gems/2.1.0/gems/fluentd-0.10.56/lib/fluent/process.rb:251:in `run'
2014-12-03 09:08:36 +0000 [error]: error on forwerder thread error="Broken pipe"
  2014-12-03 09:08:36 +0000 [error]: /usr/local/lib/ruby/gems/2.1.0/gems/fluentd-0.10.56/lib/fluent/process.rb:253:in `write'
  2014-12-03 09:08:36 +0000 [error]: /usr/local/lib/ruby/gems/2.1.0/gems/fluentd-0.10.56/lib/fluent/process.rb:253:in `to_msgpack'
  2014-12-03 09:08:36 +0000 [error]: /usr/local/lib/ruby/gems/2.1.0/gems/fluentd-0.10.56/lib/fluent/process.rb:253:in `block in run'
  2014-12-03 09:08:36 +0000 [error]: /usr/local/lib/ruby/gems/2.1.0/gems/fluentd-0.10.56/lib/fluent/process.rb:251:in `each'
  2014-12-03 09:08:36 +0000 [error]: /usr/local/lib/ruby/gems/2.1.0/gems/fluentd-0.10.56/lib/fluent/process.rb:251:in `run'

What could I be doing wrong? Can I use dynamodb-local with this plug-in?

[gonsuke]

I've never used dynamodb-local before, so I'm not sure about this issue. I'll try to reproduce this in my environment. Please give me some time.

[antohaby]

Hi, I got the same issue with UnknownError: 'uninitialized constant Fluent::DynamoDBOutput::AWS' And I have fixed it! The problem is that ruby cannot found AWS namespace, so we need to require it. First of all check: gem list | grep aws-sdk I have installed following gems: aws-sdk (2.0.33) aws-sdk-core (2.0.33) aws-sdk-resources (2.0.33) aws-sdk-v1 (1.63.0)

So here is a latest aws-sdk and old aws-sdk-v1 ( I use it for s3 plugin ) So I just change out_dynamodb.rb at 15 line i changed this require 'aws-sdk' on that require 'aws-sdk-v1' And It's working fine! I hope It will help someone.

Best Regards, Anton

[manasagn]

Do we need to install ruby to fix this issue? I am getting UnknownError: 'uninitialized constant Fluent::DynamoDBOutput::AWS' When I try to run gem list | grep aws-sdk. I get gem command not found error.

How to resolve the error without installing gem/ruby. based on error -11-22 05:36:55 -0500 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/process.rb:269:in `write'

If I do cd of /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems i see bin folder. upon doing export $PATH. still I get gem not found.