ahmedammar
commented
8 years ago This is because of #9
Open bretd25 opened 8 years ago
ahmedammar
commented
8 years ago This is because of #9
bretd25
commented
8 years ago Thank you for the response. I am now getting this error. I used this to get this far
https://github.com/genesi/logstash-input-dynamodb/tree/b1be5b5e848d53e5e10a5b72740fc4bbf9c92e94
Thanks for your help so far.
Plugin version conflict, aborting ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-core": In snapshot (Gemfile.lock): logstash-core (= 2.2.0)
In Gemfile: logstash-output-http (>= 0) java depends on logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-rabbitmq (>= 0) java depends on
logstash-core (< 3.0.0, >= 2.0.0) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-rabbitmq (>= 0) java depends on
logstash-core (< 3.0.0, >= 2.0.0) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-rabbitmq (>= 0) java depends on
logstash-core (< 3.0.0, >= 2.0.0) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-rabbitmq (>= 0) java depends on
logstash-core (< 3.0.0, >= 2.0.0) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-rabbitmq (>= 0) java depends on
logstash-core (< 3.0.0, >= 2.0.0) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-rabbitmq (>= 0) java depends on
logstash-core (< 3.0.0, >= 2.0.0) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-output-http (>= 0) java depends on
logstash-mixin-http_client (< 3.0.0, >= 2.2.0) java depends on
logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-filter-dynamodb (>= 0) java depends on
logstash-core (< 2.0.0, >= 1.4.0) java
logstash-core (= 2.2.0) java
brett--anderson
commented
8 years ago I'm getting the same problem, using jruby 1.7.20 (1.9.3p551), java version "1.8.0_66", logstash 2.2.0.
I've tried using the master version of the plugin, as well as the version forked at genesi by @ahmedammar . I'm getting the exact same errors as @bretd25
Is there a simple way to get around this? Should I just use an older version of logstash until this plugin is working with the latest?
bretd25
commented
8 years ago @brett--anderson I ended up getting the plugin installed and working even with these errors. I looked at this https://hub.docker.com/r/mantika/logstash-dynamodb-streams/~/dockerfile/
I ran this
gem install logstash-input-dynamodb logstash-filter-dynamodb
then i ran
plugin install logstash-input-dynamodb logstash-filter-dynamodb
Then i checked the list of plugins and I noticed the plugin was now in the list so i configured it and everything i working.
Hope this helps
brett--anderson
commented
8 years ago Thanks @bretd25. I'm using ElasticSearch Service on AWS which is still using version 1.5.x under the hood. This means I don't need to use LogStash 2.x and when I tried setting things up with an < 2.0 version of LogStash I got everything working.
I'll revisit this when AWS supports ES >= 2.0 and I need to update LogStash and the DynamoDB plugin
brett--anderson
commented
8 years ago @bretd25 and @ahmedammar, now that I've got this working, the raw messages from the DynamoDB stream are just being stored in ES. Do either of you know a good resource that covers setting up a LogStash filter so that each column in the original DynamoDB table is a field in the ES index?
bretd25
commented
8 years ago brett--anderson
commented
8 years ago @bretd25 Thanks for that, I implemented that filter and my table columns are now searchable document fields in the index. I have however noticed that when I restart Logstash it re-adds all the records again. I was hoping some magic would only insert changed data (and also remove and update records as needed). I'd really appreciate it if you let me know how you managed to get that to work?
bretd25
commented
8 years ago @brett--anderson
Use the perform_scan under dynamodb
dynamodb { perform_scan => "true" // this is default setting if you do not specify. This will do the initial load of table }
dynamodb { perform_scan => "false" // this will not load all records again on restart of service }
Bret
brett--anderson
commented
8 years ago @bretd25 Thanks again for your help. I got the impression that you could use Logstash to synchronise your DynamoDB and Elastic Search cluster. If I remove an item from DynamoDB, it adds an entry to ES, detailing the remove, but I expect it to actually remove the item from ES. Likewise if I stop Logstash, add a single record in DynamoDB, then restart Logstash, I would expect it to add that single record, and not scan the entire Database, adding all the records as duplicates.
Do you know if Logstash is actually a good fit for my intended scenario? I got the impression it was from the press releases from AWS, but none of their tutorials seem to go far enough to actually provide this synchronisation I require, they just log all your DynamoDB activity.
Do I need to explicitly define the primary key or something so that Logstash better understand uniqueness? Or do I need to use a custom output plugin that will handle updates and deletes in the ES cluster correctly?
marcosnils
commented
8 years ago @brett--anderson the current ES output plugin supports deleting documents, but you need to specifically configure the delete action to those streams that you want to delete:
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
I haven't done this myself but it shouldn't be too complicated.
bretd25
commented
8 years ago @brett--anderson
We do not delete any documents yet so i cannot comment on how to do it. I may need to figure this out myself though after my next meeting. If you figure it out i would love an update or if i figure out first i will update the thread.
brett--anderson
commented
8 years ago I changed my output to set a document id:
output { elasticsearch { host => "search_index.es.amazonaws.com" document_id => "%{[keys][item_uuid]}" protocol => "http" port => "80" } stdout { } }
Here 'item_uuid' is the name of my primary key in DynamoDB. The table I'm tracking only has a hash key and no range key so this is fairly straight forward. This stopped duplicates being entered into ES even if the 'perform_scan' option is enabled.
Although this stopped the duplicates, it's still logging the remove operations and presumably it would do the same for updates. I think I need to change the output definition to be conditional on the event type, and then use the correct action on the index accordingly. I believe this is what @marcosnils was referring to. I'll try and get that working and update this thread if I figure it out. Likewise, if you figure this out first or know a better way to do it, please update the thread.
marcosnils
commented
8 years ago @brett--anderson the default action of the ES output plugin is to index documents, so it won't delete anything until you change it to delete. Check the link I've posted in my previous comment.
brett--anderson
commented
8 years ago Thanks @marcosnils, I see the action parameter in that link you provided. I'm just trying to figure out how to conditionally set it, since some items may require action to be delete, while others require index.
brett--anderson
commented
8 years ago This is almost working. No duplicate documents, single deletes are propagating across to the index. I'm using Logstash 1.5.6 as I'm still having no luck with 2.2 ( @bretd25 you mentioned you got it working based on that docker file, but as far as I can tell it's specifying logstash 1.5.6?)
Although the below configuration works for single deletes, if I try and delete 3 or 4 items at a time, only around half are removed from the index. I don't know if this is something to do with trying to bulk delete over http with 1.5.6 or just a miss configuration. Anyway, it's getting closer to what I want...
input {
dynamodb{
endpoint => "dynamodb.us-west-2.amazonaws.com"
streams_endpoint => "streams.dynamodb.us-west-2.amazonaws.com"
view_type => "new_and_old_images"
aws_access_key_id => "access"
aws_secret_access_key => "secret"
table_name => "companies"
}
}
filter {
dynamodb {
}
}
output {
if [eventName] == "REMOVE" {
elasticsearch {
host => "search-endpoint.es.amazonaws.com"
document_id => "%{[keys][company_uuid]}"
protocol => "http"
port => "80"
index => "companies"
action => "delete"
}
stdout { }
} else {
elasticsearch {
host => "search-endpoint.es.amazonaws.com"
document_id => "%{[keys][company_uuid]}"
protocol => "http"
port => "80"
index => "companies"
action => "index"
}
stdout { }
}
}@brett--anderson you won't be able to run it in logstash 2.x beacuse the dynamo-filter plugin is not compatible with that version yet. I'm the author of the original plugin and I don't have the time at this moment to update it.
@bretd25 told me he was very close to get the filter plugin working in 2.x, so I guess we should wait from him.
bretd25
commented
8 years ago @marcos I do have the filter working in 2.x. Works nicely.
marcosnils
commented
8 years ago @bretd25 amazing. Any changes you can submit a PR so I can upload the new gem?
bretd25
commented
8 years ago @marcos I made no changes. Just got it installed. I have to create another environment so I will document what I did to get it installed.
marcosnils
commented
8 years ago @bretd25 amazing. We can go ahead and create a logsts 2.x docker image with both plugins installed now :D
ahmedammar
commented
8 years ago @marcosnils I'm interested to know is this docker image public? @bretd25 I'm guessing the 2.0 support I added for input is working for you?
marcosnils
commented
8 years ago Yeah, you can find the current version here: https://hub.docker.com/r/mantika/logstash-dynamodb-streams/. I'll push the 2.x tag once it's ready.
On Fri, Feb 19, 2016 at 6:33 AM, Ahmed Ammar notifications@github.com wrote:
@marcosnils https://github.com/marcosnils I'm interested to know is this docker image public? @bretd25 https://github.com/bretd25 I'm guessing the 2.0 support I added is working for you?
— Reply to this email directly or view it on GitHub https://github.com/awslabs/logstash-input-dynamodb/issues/12#issuecomment-186135507 .
bretd25
commented
8 years ago @ahmedammar Yes sorry for the delayed response.
bretd25
commented
8 years ago @marcosnils Logstash runs great if i manually run it. I am trying to get it to run as a service and i cannot get past this error message.
Sending logstash logs to /etc/logstash/logs//logstash.log. The error reported is:
you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` commandno such file to load -- com/amazonaws/aws-java-sdk-elasticbeanstalk/1.10.11/aws-java-sdk-elasticbeanstalk-1.10.11 (LoadError)
Have you seen this error before?
Bret
brett--anderson
commented
8 years ago Hi @marcosnils, any chance you could push that 2.x Logstash / DynamoDB version to Docker Hub (or even push it to a branch of the associated git repo)?
I'm about to try and get it working in Docker myself but I'd hate to waste time reinventing the wheel if you've already done this.
marcosnils
commented
8 years ago I'll push it later today
sent from mobile On Mar 10, 2016 10:03 PM, "Brett Anderson" notifications@github.com wrote:
Hi @marcosnils https://github.com/marcosnils, any chance you could push that 2.x Logstash / DynamoDB version to Docker Hub (or even push it to a fork of the associated git repo)?
I'm about to try and get it working in Docker myself but I'd hate to waste time reinventing the wheel if you've already done this.
— Reply to this email directly or view it on GitHub https://github.com/awslabs/logstash-input-dynamodb/issues/12#issuecomment-195123834 .
brett--anderson
commented
8 years ago @marcosnils Thanks! I really appreciate your work on these plugins.
I noticed you've pushed a version 1 specific Docker file to Docker Hub. It also looks like you are just in the process of deploying version 2.x Gems, is this in preparation for a version 2 Docker file? If you are actively developing these, perhaps I could throw in another question...
When I try and run the :1 Docker file with a local version of DynamoDB, I receive a "Cannot do operations on a non-existent table" error. I'm using the same access_key_id that I specified when I created the tables, and indeed using the local DynamoDB shell with that same access_key_id describes the table in question without any problems. Just in case there was still an access key id issue I tried running the local DynamoDB instance with the -sharedDb option. When I tried to run the Docker file again I got a different error: "No region found with any service for endpoint http://192.168.59.103:8000/" (Note that I'm developing in OSX using docker-osx-dev, so the containers are running inside a VM, thus the 192.~ IP address and not localhost). Again, using these same endpoints in a Python Notebook or DynamoDB shell have no problem connecting.
Does the Docker image work with a local version of DynamoDB for you? If so, any idea why I am seeing these errors? As far as I can tell the local DynamoDB instance creates the streams locally as well, as I'm able to include a stream specification when I create the table and I'm also able to then list the stream for that table and describe it as well. If you haven't tested it locally, any chance you could try it out, in case its something that can be fixed quickly? Sorry to ask so much, unfortunately I don't have the time to figure out how to setup and debug the Ruby components locally right now, so if it's relatively quick for you to do, I'd be forever grateful.
marcosnils
commented
8 years ago @brett--anderson In order to make it work with DynamoDB local you need to do some logstash configuration first. Check https://github.com/awslabs/logstash-input-dynamodb/issues/10 for more details.
Regarding version upgrade, I'm struggling with different jruby / logstash / dependencies version in order to get everything on track for version 2. I've taggeed the :1 version which is the current working version with logstash 1.5.6.
Regarding the non existing table issue, I'm not sure why it might be happening. I'm using the :1 docker image in prod without any problems. Check if you can make it work locally with the fix I've posted before and we can see from there.
brett--anderson
commented
8 years ago @marcosnils Amazing! The solution for #10 worked perfectly! For the record I ran the command:
docker run -v ~/regions.xml:/opt/logstash/regions.xml -e JAVA_OPTS=-Dcom.amazonaws.regions.RegionUtils.fileOverride=/opt/logstash/regions.xml mantika/logstash-dynamodb-streams:1 -e '
input {
dynamodb{
endpoint => "http://192.168.59.103:8000"
streams_endpoint => "http://192.168.59.103:8000"
view_type => "new_and_old_images"
aws_access_key_id => "xxx"
aws_secret_access_key => "xxx"
table_name => "companies"
}
}
filter {
dynamodb {
}
}
output {
elasticsearch {
host => "192.168.59.103"
protocol => "http"
port => "9200"
index => "companies"
}
stdout { }
}'Also for the record, I used the following regions.xml file:
<Region>
<Name>dynamodb</Name>
<Endpoint>
<ServiceName>dynamodb</ServiceName>
<Http>true</Http>
<Https>false</Https>
<Hostname>192.168.59.103</Hostname>
</Endpoint>
<Endpoint>
<ServiceName>streams.dynamodb</ServiceName>
<Http>true</Http>
<Https>false</Https>
<Hostname>192.168.59.103</Hostname>
</Endpoint>
</Region>I'm happy working with the :1 version for the moment since it works in all cases except a multi-delete which I mentioned earlier. I'll certainly upgrade to the :2 image when it become available though. Thanks again!
marcosnils
commented
8 years ago I'm happy working with the :1 version for the moment since it works in all cases except a multi-delete which I mentioned earlier
Happy to merge PR's :smile: .
2.0 is giving me a headache as it's a pain in the ass to plugins without deploying them to rubygems (logstash sucks for this). Hope to get it ready ASAP.
marcosnils
commented
8 years ago @brett--anderson @ahmedammar @bretd25 the docker image is ready:
Having a problem installing the plugin on a fresh system.
Installing logstash-input-dynamodb Plugin version conflict, aborting ERROR: Installation Aborted, message: Bundler could not find compatible versions for gem "logstash-core": In snapshot (Gemfile.lock): logstash-core (= 2.2.0)
In Gemfile: logstash-codec-cloudtrail (>= 0) java depends on logstash-codec-spool (>= 0) java depends on logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-codec-cloudtrail (>= 0) java depends on logstash-codec-spool (>= 0) java depends on logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-codec-nmap (>= 0) java depends on logstash-core (< 3.0.0, >= 2.0.0) java
logstash-codec-cloudtrail (>= 0) java depends on logstash-codec-spool (>= 0) java depends on logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-codec-cloudtrail (>= 0) java depends on logstash-codec-spool (>= 0) java depends on logstash-core (< 3.0.0, >= 2.0.0.beta2) java
logstash-codec-cloudtrail (>= 0) java depends on logstash-codec-spool (>= 0) java depends on logstash-core (< 3.0.0, >= 2.0.0.beta2) java