search

amazon-archives / logstash-input-dynamodb

This input plugin for Logstash scans a specified DynamoDB table and then reads changes to a DynamoDB table from the associated DynamoDB Stream.This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline. This gem is not a stand-alone program.
Apache License 2.0
105 stars 27 forks source link

Error while Running the Logstash Plugin #11

Open imewish opened 8 years ago

imewish commented 8 years ago

bin/logstash -f logstash-dynamodb.conf The error reported is: 

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

no such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)

Helps would be appreciated

teebu commented 8 years ago

Same issue:

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

no such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)
marcosnils commented 8 years ago

Hi,

You can check how I install the plugins in a docker container as a reference to install them in your own environment.

Logstash 1.x: https://github.com/mantika/dockerfiles/blob/1.x/logstash/dynamo-streams/Dockerfile

Logstash 2.x: https://github.com/mantika/dockerfiles/blob/master/logstash/dynamo-streams/Dockerfile

Hope it helps.

teebu commented 8 years ago

As far as I can tell, I just need to run:

RUN gem install logstash-input-dynamodb:'< 2' logstash-filter-dynamodb:'< 2'
RUN plugin install logstash-input-dynamodb logstash-filter-dynamodb

I can see logstash-input-dynamodb in my /opt/logstash/bin/plugin list It says installation successful, but when I run my test conf, same error.

marcosnils commented 8 years ago

@teebu keep in mind the stuff I do with the .m2 repository. Take into account that logstash usually runs with the user logstash user and if you install the gems / plugins using a different user you'll probably get the error you're seeing.

teebu commented 8 years ago

So after cloning the GIT, compiling, installing and uninstall the GEM a few dozen times, then doing a plugin/install I finally managed to get it to work. At one point, it only ran with SUDO, (it could have been when I SUDO gem install, instead of regular gem install) but it seems if i run it from inside /opt/logstash folder, it is working.

Now, it seems I can run it without needing to be in the folder. With no errors.

I think the step where I had to compile might have installed those missing JAR files.

imewish commented 8 years ago

@marcosnils This is Awesome ..thanks for updating the logstash gem with this dynamodb plugin. It woks fine. Installing this locally and works properly was a nightmare, it was needed to follow wired ways to make it work. Thanks again

teebu commented 8 years ago

I am still having a problem. Even though I was able to run the plugin, i was not able to run it with the logstash service.

Getting error: :timestamp=>"2016-03-19T07:35:37.371000+0000", :message=>"The error reported is: \n \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars withlock_jarscommand\n\nno such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)"}

I tried to follow your docker example, but I dont understand what the .m2 sym link is doing. Here is my folder structure:

I am on an EC2 machine.

home: /home/ec2-user .m2 folder is inside: /home/ec2-user/.m2 the com/aws files are listed in: /home/ec2-user/.m2/repository

Doing ln -s /var/lib/logstash/.m2 /root/.m2 doesn't work for me. I tried ln -s /var/lib/logstash/.m2 ~/.m2 looks like it creates a sym link in .m2 folder to /var/lib/logstash/.m2, so thats probably wrong.

I really need a step by step guide, because I just spend 6 hours trying to get this to work.

Right now trying to run a config, I get the error no such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError) unless I run it with sudo.

marcosnils commented 8 years ago

Install jRuby

cd /etcsudo mkdir jrubysudo wget https https://s3.amazonaws.com/jruby.org/downloads/9.0.5.0/jruby-bin-9.0.5.0.tar.gz :// https://s3.amazonaws.com/jruby.org/downloads/9.0.5.0/jruby-bin-9.0.5.0.tar.gz s3.amazonaws.com https://s3.amazonaws.com/jruby.org/downloads/9.0.5.0/jruby-bin-9.0.5.0.tar.gz / https://s3.amazonaws.com/jruby.org/downloads/9.0.5.0/jruby-bin-9.0.5.0.tar.gz jruby.org https://s3.amazonaws.com/jruby.org/downloads/9.0.5.0/jruby-bin-9.0.5.0.tar.gz / https://s3.amazonaws.com/jruby.org/downloads/9.0.5.0/jruby-bin-9.0.5.0.tar.gz downloads https://s3.amazonaws.com/jruby.org/downloads/9.0.5.0/jruby-bin-9.0.5.0.tar.gz /9.0.5.0/ https://s3.amazonaws.com/jruby.org/downloads/9.0.5.0/jruby-bin-9.0.5.0.tar.gz jruby-bin-9.0.5.0.tar.gz https://s3.amazonaws.com/jruby.org/downloads/9.0.5.0/jruby-bin-9.0.5.0.tar.gzsudo tar -xvf jruby-bin-9.0.5.0.tar.gzexport PATH=/etc/jruby/jruby-9.0.5.0/bin:$PATH$Test to make sure its installed correctlyjruby -v

Install Git

sudo yum install git

Install Maven

Sudo wget http:// http://apache.spinellicreations.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz apache.spinellicreations.com http://apache.spinellicreations.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz / http://apache.spinellicreations.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz maven http://apache.spinellicreations.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz /maven-3/3.3.9/ http://apache.spinellicreations.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz binaries http://apache.spinellicreations.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz / http://apache.spinellicreations.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz apache-maven-3.3.9-bin.tar.gz http://apache.spinellicreations.com/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gzsudo tar -xvf apache-maven-3.3.9-bin.tar.gzexport PATH=/etc/maven/apache-maven-3.3.9/bin:$PATH$Test to make sure its installed correctlymvn -v

Install Logstash

sudo rpm --import https https://packages.elastic.co/GPG-KEY-elasticsearch :// https://packages.elastic.co/GPG-KEY-elasticsearchpackages.elastic.co https://packages.elastic.co/GPG-KEY-elasticsearch/ https://packages.elastic.co/GPG-KEY-elasticsearchGPG-KEY-elasticsearch https://packages.elastic.co/GPG-KEY-elasticsearchsudo nano logstash.repo [logstash-2.2]

name=Logstash repository for 2.2.x packages

baseurl=http:// http://packages.elastic.co/logstash/2.2/centos packages.elastic.co http://packages.elastic.co/logstash/2.2/centos/ http://packages.elastic.co/logstash/2.2/centoslogstash http://packages.elastic.co/logstash/2.2/centos/2.2/ http://packages.elastic.co/logstash/2.2/centoscentos http://packages.elastic.co/logstash/2.2/centos

gpgcheck=1

gpgkey=http:// http://packages.elastic.co/GPG-KEY-elasticsearch packages.elastic.co http://packages.elastic.co/GPG-KEY-elasticsearch/ http://packages.elastic.co/GPG-KEY-elasticsearchGPG-KEY-elasticsearch http://packages.elastic.co/GPG-KEY-elasticsearch

enabled=1

sudo yum install logstashexport PATH=/opt/logstash/bin:$PATH$

Install Logstash-input-dynamodb

Clonesudo git clone https https://github.com/genesi/logstash-input-dynamodb.git:// https://github.com/genesi/logstash-input-dynamodb.gitgithub.com https://github.com/genesi/logstash-input-dynamodb.git/ https://github.com/genesi/logstash-input-dynamodb.gitgenesi https://github.com/genesi/logstash-input-dynamodb.git/ https://github.com/genesi/logstash-input-dynamodb.git logstash-input-dynamodb.git https://github.com/genesi/logstash-input-dynamodb.gitsudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/bin/sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/specifications/jruby -S bundle installsudo chmod a+wrx /etc/logstash-input-dynamodbjruby -S gem install logstash-input-dynamodb-1.0.1-java.gemsudo chmod a+wrx /opt/logstash/Gemfilesudo chmod a+wrx /etc/logstash-input-dynamodb/lib/logstash-input-dynamodb_jars.rbsudo chmod a+wrx /etc/logstash-input-dynamodb/libBundle installplugin install logstash-input-dynamodb-1.0.1-java.gemDeal with any dependenciesgem "logstash-input-dynamodb", "1.0.1", :path => "vendor/local_gems/863a3d26/logstash-input-dynamodb-1.0.1-java"gem "activesupport-json_encoder","1.1.0",:path => "/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-json_encoder"gem "activesupport" , "4.1.14.1" , :path => "/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-4.1.14.1"

Install logstash-filter-dynamodb

cd /etcgit clone https https://github.com/genesi/logstash-filter-dynamodb.git:// https://github.com/genesi/logstash-filter-dynamodb.gitgithub.com https://github.com/genesi/logstash-filter-dynamodb.git/ https://github.com/genesi/logstash-filter-dynamodb.gitgenesi https://github.com/genesi/logstash-filter-dynamodb.git/ https://github.com/genesi/logstash-filter-dynamodb.git logstash-filter-dynamodb.git https://github.com/genesi/logstash-filter-dynamodb.gitsudo chmod a+wrx /etc/logstash-filter-dynamodb/jruby -S bundle installjruby -S gem build logstash-filter-dynamodb.gemspecplugin install --no-verify /etc/logstash-filter-dynamodb/logstash-filter-dynamodb-0.0.4.gem

Fix Logstash to run as service

Copy logstash-input-dynamodb to a new locationsudo cp -r logstash-input-dynamodb-1.0.1-java/ /opt/logstash/vendor/bundle/jruby/1.9/gems/Update /opt/logstash/Gemfilegem "logstash-input-dynamodb", "1.0.1", :path => "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-java"sudo chmod a+wrx /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-javasudo chmod a+wrx /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-java/lib/logstash-input-dynamodb_jars.rbsudo chmod a+wrx /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-java/lib/cd /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-javabundle update

/opt/logstash/Gemfile bottom of file

gem "logstash-input-http_poller"

gem "logstash-input-dynamodb", "1.0.1", :path => "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb-1.0.1-java"

gem "activesupport-json_encoder", "1.1.0", :path => "/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-json_encoder"

gem "activesupport", "4.1.14.1", :path => "/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-4.1.14.1"

gem "logstash-filter-dynamodb", "0.0.4", :path => "vendor/local_gems/3182dbdf/logstash-filter-dynamodb-0.0.4"

sent from mobile On Mar 19, 2016 4:57 AM, "teebu" notifications@github.com wrote:

I am still having a problem. Even though I was able to run the plugin, i was not able to run it with the logstash service.

Getting error: :timestamp=>"2016-03-19T07:35:37.371000+0000", :message=>"The error reported is: \n \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command\n\nno such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)"}

I tried to follow your docker example, but I dont understand what the .m2 sym link is doing. Here is my folder structure:

I am on an EC2 machine.

home: /home/ec2-user .m2 folder is inside: /home/ec2-user/.m2 the com/aws files are listed in: /home/ec2-user/.m2/repository

Doing ln -s /var/lib/logstash/.m2 /root/.m2 doesn't work for me. I tried ln -s /var/lib/logstash/.m2 ~/.m2 looks like it creates a sym link in .m2 folder to /var/lib/logstash/.m2, so thats probably wrong.

I really need a step by step guide, because I just spend 6 hours trying to get this to work.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/awslabs/logstash-input-dynamodb/issues/11#issuecomment-198662999

teebu commented 8 years ago

thanks, this was very hard to read with the lack of formatting. ill give it a shot. also, im using LS 1.5

I tried to follow your steps, but it feels like there are either steps missing, or some mismatch is going on. I have maven,jruby installed, with LS 1.5.

teebu commented 8 years ago

I managed to translate whatever you wrote to this:

Clone 
sudo git clone https://github.com/genesi/logstash-input-dynamodb.git
sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/
sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/
sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/bin/
sudo chmod a+wrx /etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/specifications/
jruby -S bundle install
sudo chmod a+wrx /etc/logstash-input-dynamodb

did you miss a step here? to build the gem?

jruby -S gem install logstash-input-dynamodb-1.0.1-java.gem
sudo chmod a+wrx /opt/logstash/Gemfile
sudo chmod a+wrx /etc/logstash-input-dynamodb/lib/logstash-input-dynamodb_jars.rb
sudo chmod a+wrx /etc/logstash-input-dynamodb/lib
Bundle install
plugin install logstash-input-dynamodb-1.0.1-java.gem

Around this point, there is no "Bundle" so I fix for that, to make it "bundle"

plugin install doesn't work without sudo. I fix for that.

Doing: sudo bin/plugin install /etc/logstash-input-dynamodb/logstash-input-dynamodb-1.0.0-java.gem

Getting Error: An error occurred while installing logstash-input-dynamodb (1.0.0), and Bundler cannot continue. Make sure that gem install logstash-input-dynamodb -v '1.0.0' succeeds before bundling.

Trying: sudo gem install logstash-input-dynamodb -v '1.0.0' inside /opt/logstash folder. -- Successfully installed logstash-input-dynamodb-1.0.0-java 1 gem installed

Still getting same error when doing plugin install: An error occurred while installing logstash-input-dynamodb (1.0.0), and Bundler cannot continue. Make sure thatgem install logstash-input-dynamodb -v '1.0.0'succeeds before bundling.

Deal with any dependencies 
gem "logstash-input-dynamodb", "1.0.1", :path => "vendor/local_gems/863a3d26/logstash-input-dynamodb-1.0.1-java"
gem "activesupport-json_encoder","1.1.0",:path => "/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-json_encoder"
gem "activesupport" , "4.1.14.1" , :path => "/etc/jruby/jruby-9.0.5.0/lib/ruby/gems/shared/gems/activesupport-4.1.14.1"

Fix Logstash to run as service

Copy logstash-input-dynamodb to a new location
sudo cp -r logstash-input-dynamodb-1.0.1-java/ /opt/logstash/vendor/bundle/jruby/1.9/gems/
Update /opt/logstash/Gemfile
gem "logstash-input-dynamodb", "1.0.0", :path => "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb"
sudo chmod a+wrx /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb
sudo chmod a+wrx /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb/lib/logstash-input-dynamodb_jars.rb
sudo chmod a+wrx /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb/lib/
cd /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb
bundle update

One thing I noticed was changing the owner of 'opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-dynamodb' to logstash seems to be doing something

One thing to note: Using the rubygems.org version, '1.0.3' by doing just sudo bin/plugin install logstash-input-dynamodb, throws error: {:timestamp=>"2016-03-19T20:37:11.323000+0000", :message=>"The error reported is: \n \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars withlock_jarscommand\n\nno such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)"} when starting logstash service.

marcosnils commented 8 years ago

did you miss a step here? to build the gem?

Probably.

I didn't write this instructions, these were written by @bretd25. Maybe he can help to indicate how he solved the problem.

Keep in mind that this instructions are for logstash 2.x.. You'll have to adapt them to your version.

Marcos.

paragshar commented 8 years ago

I am getting this error 

you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

no such file to load -- joda-time/joda-time/2.8.2/joda-time-2.8.2 (LoadError)

No clue on how to solve.

marcosnils commented 8 years ago

Which logstash version are you using and how are you installing the gem?

paragshar commented 8 years ago

Its logstash 2.2.2, and installed the gem through gem install gem install logstash-input-dynamodb

Version for logstash-input-dynamodb is 2.0.3, got it from https://rubygems.org/gems/logstash-input-dynamodb/versions/2.0.3-java

Earlier tried the local build option, but ended up at exact same problem.

paragshar commented 8 years ago

Also I see all the required jars are right there in my ~/.m2/repository folder, either directly or in some folders. Why is logstash still not able to find them?

marcosnils commented 8 years ago

because logstash usually uses the logstash user when it starts as logstash agent. You can try to copy or symlink your .m2 folder to where logstash searches for dependencies (I believe is something like /opt/logstash)

marcosnils commented 8 years ago

you probably did gem install with your user instead of logstash's. That's why deps got installed in your local .m2 folder.

paragshar commented 8 years ago

Symlinking does not seem to help. I created a symlink named .m2 in /opt/logstash where my logstash resides. Exact same problem :( Even copying of entire .m2 folder, and changing ownership to logstash:logstash of the folder recursively does not help. Same error.

marcosnils commented 8 years ago

@paragshar try creating this dir and linking .m2 there:

mkdir -p /var/lib/logstash/.m2
ln -s /var/lib/logstash/.m2 /your/user/.m2

Hope it helps.

paragshar commented 8 years ago

Not too sure what you meant by above. Should the link part not be like

ln -s  /your/user/.m2 /var/lib/logstash/.m2

and /your/user substituted by /home/ubuntu I hope we are trying to create a softlink .m2 which points to maven repo created on /home/ubuntu

paragshar commented 8 years ago

I figured it out finally. There was no jar file for joda-time/joda-time/2.8.2/joda-time-2.8.2 in my .m2/repository/joda-time/joda-time/2.8.2. I just had a pom file there. I downloaded a jar file (joda-time-2.8.2.jar) directly and placed in within the folder /var/lib/logstash/.m2/repository/joda-time/joda-time/2.8.2.

And it works like a breeze after that.

Many thanks. Your assistance really helped me figure out the issue.

But I still need to see why was the jar file not there in the first place :)

teebu commented 8 years ago

Can you help me get rid of this error, whenever I try to install any plugin.

It installs the plugin I was trying to install successfully, but instead of finishing with a success, it does this with 1/10 retries. An error occurred while installing logstash-input-dynamodb (1.0.0), and Bundler cannot continue. Make sure thatgem install logstash-input-dynamodb -v '1.0.0'succeeds before bundling.

paragshar commented 8 years ago

The best idea is to run

gem install logstash-input-dynamodb

This will download and install the plugin from the gem repo. Let me know if you encounter any problem with this.

paragshar commented 8 years ago

https://rubygems.org/gems/logstash-input-dynamodb

teebu commented 8 years ago

i tried installing it from the gem repo before, it installs 1.0.3. but then it doesn't run in the LS service.

Using the rubygems.org version, '1.0.3' by doing just sudo bin/plugin install logstash-input-dynamodb, throws error: 

{:timestamp=>"2016-03-19T20:37:11.323000+0000", :message=>"The error reported is: \n \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- com/amazonaws/aws-java-sdk-kms/1.10.27/aws-java-sdk-kms-1.10.27 (LoadError)"}.

when starting logstash service