search

amazon-archives / logstash-input-dynamodb

This input plugin for Logstash scans a specified DynamoDB table and then reads changes to a DynamoDB table from the associated DynamoDB Stream.This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline. This gem is not a stand-alone program.
Apache License 2.0
105 stars 27 forks source link

Plugin's AWS Java SDK version not compatible with AWS ECS Agent #29

Open trollefson opened 7 years ago

trollefson commented 7 years ago

The current Java SDK version in use by the plugin is 1.10.11

On July 13th, 2016 AWS began supporting IAM roles for ECS Tasks. In order to use this feature a Java SDK version >= 1.11.16 is required.

I am running Logstash with this plugin via a Docker container on AWS ECS. Rather than passing in credentials to the plugin I let credentials be discovered through the AWS Provider Chain. My ECS Container Instance ends up discovering credentials via the instance profile credentials (step 5) rather than the Amazon ECS container credentials I have configured (step 4). This appear to be because of the outdated Java SDK used by the plugin.

Please update the Java SDK version to be >= 1.11.16.

Thanks!

marcosnils commented 7 years ago

@trollefson are you using your own docker image, or you're using this one?https://hub.docker.com/r/mantika/logstash-dynamodb-streams/

trollefson commented 7 years ago

@marcosnils That one extended with the ability to pass in custom configs

marcosnils commented 7 years ago

@trollefson ok, in that case you need to send a PR to this branch https://github.com/marcosnils/logstash-input-dynamodb/tree/deps_update with the dependency updates so I can merge and rebuild the image.

This repo is practically dead, and I'm not using logstash anymore, so I won't invest too much time on this.

trollefson commented 7 years ago

@marcosnils Thank you for your help. I will be sure to do that today

jehanzebqayyum commented 7 years ago

@marcosnils what are you using to connect dynamodb to elasticsearch ?

marcosnils commented 7 years ago

@jehanzebqayyum I'm now using dynamodb streams + AWS lambda.