amazon-archives / logstash-input-dynamodb

This input plugin for Logstash scans a specified DynamoDB table and then reads changes to a DynamoDB table from the associated DynamoDB Stream.This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline. This gem is not a stand-alone program.
Apache License 2.0
105 stars 27 forks source link

Installation issues #17

Open avtaniket opened 8 years ago

avtaniket commented 8 years ago

Hi,

During setup I can across below issues, using logstash 1.5

1> jruby -S gem install --local logstash-input-dynamodb-1.0.0-java.gem ERROR: Could not find a valid gem 'logstash-core' (< 2.0.0, >= 1.4.0) in any repository

2> sudo bin/plugin install –no-verify logstash-input-dynamodb Validating –no-verify Plugin –no-verify does not exist ERROR: Installation aborted, verification failed for –no-verify

3> sudo bin/plugin install –no-verify logstash-input-dynamodb-1.0.0.gem ERROR: Mixed source of plugins, you can't mix local .gem and remote gems

See: 'bin/plugin install --help'

ghost commented 8 years ago

having the same issue, I tried installing logstash-core and picked an arbitrary version that fell in the <2.0.0, >=1.4.0 category (i.e. 1.5.6) and that still is not working. Here's what I'm getting now:

ubuntu@ip-172-31-15-36:~/logstash-input-dynamodb$ jruby -S gem install --local logstash-input-dynamodb-1.0.0-java.gem ERROR: While executing gem ... (Gem::DependencyResolutionError) conflicting dependencies logstash-core (< 3.0.0, >= 2.0.0.beta2) and logstash-core (< 2.0.0, >= 1.4.0) Activated logstash-core-1.5.6 which does not match conflicting dependency (< 3.0.0, >= 2.0.0.beta2)

Conflicting dependency chains: logstash-input-dynamodb (= 1.0.0), 1.0.0 activated, depends on logstash-core (< 2.0.0, >= 1.4.0), 1.5.6 activated

versus: logstash-input-dynamodb (= 1.0.0), 1.0.0 activated, depends on logstash-codec-json (>= 0), 2.1.0 activated, depends on logstash-core (< 3.0.0, >= 2.0.0.beta2)

Gems matching logstash-core (< 3.0.0, >= 2.0.0.beta2): logstash-core-2.0.0.rc1-java, logstash-core-2.2.2-java, logstash-core-2.3.0.snapshot5-java

I'm not quite sure how dependencies are getting resolved but there's clearly some issues with the way dependencies are defined, that'd be my guess anyway.

devnullnor commented 8 years ago

I have the same problem. If this isn't solved soon I'm turning to DocumentDB from MS.

marcosnils commented 8 years ago

@devnullnor can't you use the docker container?

devnullnor commented 8 years ago

How can I use docker for this task? Will it cost me anything? Thank you.

avtaniket commented 8 years ago

@devnullnor Docker is free, I tried with docker and now its working perfectly.

Check link https://hub.docker.com/r/mantika/logstash-dynamodb-streams/

1> Step 1 - Install docker if you don't have already

2> Step 2 - Pull image $ docker pull mantika/logstash-dynamodb-streams

Hope this help you to get started.

devnullnor commented 8 years ago

Docker seems to have fixed it. Thank you! Question: Do I need to run docker run mantika/logstash-dynamodb-streams -e periodically, or is this a continuous stream that will pull data from Dynamo?

marcosnils commented 8 years ago

@devnullnor

No, logstash will automatically propagate the changes to elasticsearch, you don't need to run it periodically.

devnullnor commented 8 years ago

I get an error: {:timestamp=>"2016-04-07T09:59:15.139000+0000", :message=>"Connection refused", :class=>"Manticore::SocketException", :level=>:error} The error reported is: Requested resource not found: Table: users not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException; Request ID: 81F1UHJKBPLIP8GC630S19274JVV4KQNSO5AEMVJF66Q9ASUAAJG)

Which connection is refused?

shikasta-kashti commented 8 years ago

I'm having the same issue on Windows 7. Any updates?

C:\logstash-input-dynamodb (master) λ jruby -S gem install --local logstash-input-dynamodb-1.0.0-java.gem ERROR: Could not find a valid gem 'logstash-core' (< 2.0.0, >= 1.4.0) in any repository

marcosnils commented 8 years ago

@mwahidje https://github.com/awslabs/logstash-input-dynamodb/issues/21

shikasta-kashti commented 8 years ago

Thanks. It seems to be working now.