search

connamara / logstash-filter-fix_protocol

FIX Protocol Logstash Filter
Other
42 stars 27 forks source link

Cannot Start Logstash w/ Fix Protocol Filter #65

Closed sking2 closed 8 years ago

sking2 commented 8 years ago

Followed installation instructions and results in inability to load logstash config:

[michael@uat-service1.chi ~]$ tail -f /var/log/logstash/logstash.log {:timestamp=>"2016-04-04T08:20:29.668000-0700", :message=>"fetched an invalid config", :config=>"input {\n file {\n path => \"/motif/logs/tradeexecution.console.log\"\n start_position => \"beginning\"\n }\n}\nfilter {\n grok {\n match => [\"message\",\"%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:fix_session}: %{GREEDYDATA:fix_string}\"]\n }\n fix_protocol {\n fix_message => fix_string\n }\n}\noutput {\n amazon_es {\n hosts => [\"search-fix-logs-ht5lnwidn2x4e6g7czxxqdbrw4.us-east-1.es.amazonaws.com\"]\n region => \"us-east-1\"\n index => \"uat-fix-logs-%{+YYYY.MM.dd}\"\n\t aws_access_key_id => 'ACCESS_KEY'\n aws_secret_access_key => 'SECRET_KEY'\n }\n}\n\n", :reason=>"Couldn't find any filter plugin named 'fix_protocol'. Are you sure this is correct? Trying to load the fix_protocol filter plugin resulted in this error: uninitialized constant ActiveSupport::Autoload", :level=>:error}

daino3 commented 8 years ago

Hello. What version of logstash are you running? And - just to make sure - you've installed the plugin with the command below, correct?

$ /opt/logstash/bin/plugin install logstash-filter-fix_protocol
sking2 commented 8 years ago

Used the following to install:

created file logstash.repo : [logstash-2.3] name=Logstash repository for 2.3.x packages baseurl=http://packages.elastic.co/logstash/2.3/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1

sudo cp ~/logstash.repo /etc/yum.repos.d/logstash.repo sudo rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch sudo yum -y install logstash sudo /opt/logstash/bin/logstash-plugin install logstash-output-amazon_es sudo /opt/logstash/bin/logstash-plugin install logstash-filter-fix_protocol

daino3 commented 8 years ago

Can you paste the output from 

sudo /opt/logstash/bin/logstash-plugin install logstash-filter-fix_protocol

I just had trouble myself installing the gem with logstash 2.3, but everything works as expected for logstash 2.2. We'll need to dig into changes introduced between versions.

sking2 commented 8 years ago

Just downgraded to 2.2 and still doesnt work:

{:timestamp=>"2016-04-04T13:33:04.022000-0700", :message=>"An unexpected error occurred!", :error=>#<LogStash::PluginLoadingError: Couldn't find any filter plugin named 'fix_protocol'. Are you sure this is correct? Trying to load the fix_protocol filter plugin resulted in this error: uninitialized constant ActiveSupport::Autoload>, :class=>"LogStash::PluginLoadingError", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.3-java/lib/logstash/plugin.rb:85:in lookup'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.3-java/lib/logstash/pipeline.rb:400:inplugin'", "(eval):36:in initialize'", "org/jruby/RubyKernel.java:1079:ineval'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.3-java/lib/logstash/pipeline.rb:57:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.3-java/lib/logstash/agent.rb:166:inexecute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.3-java/lib/logstash/runner.rb:90:in run'", "org/jruby/RubyProc.java:281:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.3-java/lib/logstash/runner.rb:95:in run'", "org/jruby/RubyProc.java:281:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:in `initialize'"], :level=>:warn}

The message changed slightly, and is now a WARN, but still doesnt work

sking2 commented 8 years ago

[michael@uat-service1.chi ~]$ sudo /opt/logstash/bin/plugin install logstash-filter-fix_protocol Validating logstash-filter-fix_protocol Installing logstash-filter-fix_protocol Installation successful

daino3 commented 8 years ago

I can't reproduce the issue you're seeing. My only thought is your gem pathing for logstash is screwy and the plugin isn't getting installed to the appropriate directory. Can you check to see if the gem logstash-filter-fix_protocol-0.2.0 is actually in /opt/logstash/vendor/bundle/jruby/1.9/gems?

labdi24 commented 8 years ago

I too am having the same issue on both logstash 2.2.2 and 2.2.3. see below. any ideas I've been at this for a few hours and just can't seem to figure it out.

Plugin not defined in namespace, checking for plugin file {:type=>"input", :name=>"stdin", :path=>"logstash/inputs/stdin", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"} Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"line", :path=>"logstash/codecs/line", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"} config LogStash::Codecs::Line/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Codecs::Line/@delimiter = "\n" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::Stdin/@codec = <LogStash::Codecs::Line charset=>"UTF-8", delimiter=>"\n"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::Stdin/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"grok", :path=>"logstash/filters/grok", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"} config LogStash::Filters::Grok/@match = {"message"=>"%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:fix_session}: %{GREEDYDATA:fix_string}"} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@patterns_dir = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@break_on_match = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@named_captures_only = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@keep_empty_captures = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@singles = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@tag_on_failure = ["_grokparsefailure"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Filters::Grok/@overwrite = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} Plugin not defined in namespace, checking for plugin file {:type=>"filter", :name=>"fix_protocol", :path=>"logstash/filters/fix_protocol", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"} The error reported is: Couldn't find any filter plugin named 'fix_protocol'. Are you sure this is correct? Trying to load the fix_protocol filter plugin resulted in this error: uninitialized constant ActiveSupport::Autoload /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/plugin.rb:85:in lookup' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:400:inplugin'

sking2 commented 8 years ago

Its for sure there. From Googling the problem is somewhere in the dependency management inside the plugin.

[michael@uat-service1.chi ~]$ ls -l /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.2.0/ total 40 -rw-r--r-- 1 logstash logstash 1421 Apr 4 08:19 CODE_OF_CONDUCT.md -rw-r--r-- 1 logstash logstash 1278 Apr 4 08:19 CONTRIBUTING.md -rw-r--r-- 1 logstash logstash 104 Apr 4 08:19 Gemfile drwxr-xr-x 3 logstash logstash 4096 Apr 4 08:19 lib -rw-r--r-- 1 logstash logstash 563 Apr 4 08:19 LICENSE.txt -rw-r--r-- 1 logstash logstash 1366 Apr 4 08:19 logstash-filter-fix_protocol.gemspec -rw-r--r-- 1 logstash logstash 2148 Apr 4 08:19 QUICKFIX_LICENSE.txt -rw-r--r-- 1 logstash logstash 5945 Apr 4 08:19 README.md drwxr-xr-x 5 logstash logstash 4096 Apr 4 08:19 spec

labdi24 commented 8 years ago

ls -l /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.2.0/ total 40 -rw-r--r--. 1 root root 1421 Apr 4 16:42 CODE_OF_CONDUCT.md -rw-r--r--. 1 root root 1278 Apr 4 16:42 CONTRIBUTING.md -rw-r--r--. 1 root root 104 Apr 4 16:42 Gemfile drwxr-xr-x. 3 root root 4096 Apr 4 16:42 lib -rw-r--r--. 1 root root 563 Apr 4 16:42 LICENSE.txt -rw-r--r--. 1 root root 1366 Apr 4 16:42 logstash-filter-fix_protocol.gemspec -rw-r--r--. 1 root root 2148 Apr 4 16:42 QUICKFIX_LICENSE.txt -rw-r--r--. 1 root root 5945 Apr 4 16:42 README.md drwxr-xr-x. 5 root root 4096 Apr 4 16:42 spec

labdi24 commented 8 years ago

This is nuts, especially since I am not a developer.. Ive been at this for so long I think I am just going to give up... sking2, were you able to resolve your issue?

labdi24 commented 8 years ago

Now I'm getting the follow errors:

Plugin not defined in namespace, checking for plugin file {:type=>"input", :name=>"file", :path=>"logstash/inputs/file", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"} Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"plain", :path=>"logstash/codecs/plain", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"} config LogStash::Codecs::Plain/@charset = "UTF-8" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@path = ["/etc/logstash/conf.d/fix-input.log"] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@start_position = "beginning" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain charset=>"UTF-8"> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@stat_interval = 1 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@discover_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@sincedb_write_interval = 15 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@delimiter = "\n" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@ignore_older = 86400 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} config LogStash::Inputs::File/@close_older = 3600 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"122", :method=>"config_init"} The error reported is: wrong number of arguments calling filter (0 for 1) (eval):11:in initialize' org/jruby/RubyKernel.java:1079:ineval' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/pipeline.rb:57:in initialize' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/agent.rb:172:inexecute' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/runner.rb:90:in run' org/jruby/RubyProc.java:281:incall' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.2.2-java/lib/logstash/runner.rb:95:in run' org/jruby/RubyProc.java:281:incall' /opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:in `initialize'

daino3 commented 8 years ago

@labdi24 - I will look into this tomorrow. Do me a favor though please and run the command below. Then, let me know if the plugin is even listed in the output. 

sudo /opt/logstash/bin/plugin list
labdi24 commented 8 years ago

A short list of plugins. /opt/logstash/bin/plugin list .... logstash-filter-drop logstash-filter-fingerprint logstash-filter-fix_protocol ....

Not sure if it makes any difference, changed the logstash.conf on the line where it calls the filter plugin from "fix_protocol" to "logstash-filter-fix_protocol". I no longer receive the error "Can not find plugin". It appears logstash is unable to reference the plugin by "fix_protocol". Thanks for your help.

labdi24 commented 8 years ago

Hi Daino,

Any ideas?

daino3 commented 8 years ago

@labdi24 - Sort of a shot in the dark given I can't reproduce the issue, but I'm hoping that #66 fixes your guys' issues. Update the gem to 0.2.1 with 

/opt/logstash/bin/plugin update logstash-filter-fix_protocol

and let me know if that solves your issue.

Also, you shouldn't need to change the name of the filter in your .conf file. So revert that change before firing up logstash. Good luck!

daino3 commented 8 years ago

@labdi24 & @sking2 - once you get a chance to try out the fix, can you reply back and šŸ‘ or šŸ‘Ž so I can close the issue? Thanks!

labdi24 commented 8 years ago

Hi Daino3,

Voila, it logstash now starts. success! Thanks again.