Plugin broken after LS upgrade

[Paraomao]

After updating LS to 5.4.3 and updating all LS plugins, the dissect plugin does not load anymore.

Logstash log the error message reported below:

[2017-06-28T16:47:07,348][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Couldn't find any filter plugin named 'dissect'. Are you sure this is correct? Trying to load the dissect filter plugin resulted in this error: Problems loading the requested plugin named dissect of type filter. Error: NameError NameError"} [2017-06-28T16:47:31,978][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"dissect", :path=>"logstash/filters/dissect", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:221:in namespace_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:157:inlegacy_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:133:in lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:175:inlookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:139:in lookup'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:91:inplugin'", "(eval):3616:in initialize'", "org/jruby/RubyKernel.java:1079:ineval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:63:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:145:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:286:in create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:inregister_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:274:in execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

The list of the installed plugins is reported below:

logstash-codec-cloudfront (3.0.1) logstash-codec-cloudtrail (3.0.2) logstash-codec-json (3.0.3) logstash-codec-json_lines (3.0.3) logstash-codec-line (3.0.3) logstash-codec-multiline (3.0.5) logstash-codec-plain (3.0.3) logstash-codec-rubydebug (3.0.3) logstash-filter-clone (3.0.3) logstash-filter-date (3.1.7) logstash-filter-de_dot (1.0.1) logstash-filter-dissect (1.0.9) logstash-filter-drop (3.0.3) logstash-filter-fingerprint (3.0.4) logstash-filter-geoip (4.2.1) logstash-filter-grok (3.4.2) logstash-filter-json (3.0.3) logstash-filter-kv (4.0.1) logstash-filter-mutate (3.1.5) logstash-filter-ruby (3.0.3) logstash-filter-split (3.1.2) logstash-filter-syslog_pri (3.0.3) logstash-filter-throttle (4.0.2) logstash-filter-useragent (3.1.1) logstash-filter-uuid (3.0.3) logstash-input-beats (4.0.4) logstash-input-exec (3.1.3) logstash-input-file (4.0.2) logstash-input-generator (3.0.3) logstash-input-heartbeat (3.0.3) logstash-input-http_poller (3.3.1) logstash-input-lumberjack (3.1.2) logstash-input-pipe (3.0.4) logstash-input-stdin (3.2.3) logstash-input-syslog (3.2.1) logstash-input-tcp (4.2.2) logstash-output-elasticsearch (7.3.6) logstash-output-email (4.0.4) logstash-output-exec (3.1.1) logstash-output-file (4.0.2) logstash-output-http (4.3.1) logstash-output-lumberjack (3.1.3) logstash-output-null (3.0.3) logstash-output-pipe (3.0.3) logstash-output-stdout (3.1.1) logstash-patterns-core (4.1.1)

I am using Centos 7

To reproduce the failing scenario: Install LS 5.4.3 Install the dissect filter Update all LS plugins Configure a LS filter with dissect Start LS and look at the logs.

BR

[ThomasADavis]

I am seeing this also.

[Paraomao]

The issue is present also using LS 5.5.0

[ThomasADavis]

I also tried this:

/usr/share/logstash/bin/logstash-plugin install --version 1.0.6 logstash-filter-dissect

and it still broke, so I'm not 100% sure it's the plugin.

[untergeek]

Why are you trying to install the dissect filter? It is bundled...

root@logstash:~# ls /usr/share/logstash/vendor/bundle/jruby/1.9/gems/| grep diss
logstash-filter-dissect-1.0.8
root@logstash:~# /usr/share/logstash/bin/logstash --version
logstash 5.4.3

[untergeek]

I upgraded from 5.4.0 to 5.4.3 and dissect just carried over for me (I didn't install it manually on 5.4.0 either). It works in my configuration.

[untergeek]

@Paraomao that list of installed plugins seems, well, short. This is a default DEB package install, plus x-pack:

root@logstash:~# /usr/share/logstash/bin/logstash-plugin list --verbose
logstash-codec-cef (4.1.2)
logstash-codec-collectd (3.0.4)
logstash-codec-dots (3.0.2)
logstash-codec-edn (3.0.2)
logstash-codec-edn_lines (3.0.2)
logstash-codec-es_bulk (3.0.3)
logstash-codec-fluent (3.1.1)
logstash-codec-graphite (3.0.2)
logstash-codec-json (3.0.2)
logstash-codec-json_lines (3.0.2)
logstash-codec-line (3.0.2)
logstash-codec-msgpack (3.0.2)
logstash-codec-multiline (3.0.3)
logstash-codec-netflow (3.4.0)
logstash-codec-plain (3.0.2)
logstash-codec-rubydebug (3.0.2)
logstash-filter-clone (3.0.2)
logstash-filter-csv (3.0.3)
logstash-filter-date (3.1.5)
logstash-filter-dissect (1.0.8)
logstash-filter-dns (3.0.3)
logstash-filter-drop (3.0.2)
logstash-filter-fingerprint (3.0.3)
logstash-filter-geoip (4.1.1)
logstash-filter-grok (3.4.1)
logstash-filter-json (3.0.2)
logstash-filter-kv (4.0.0)
logstash-filter-metrics (4.0.2)
logstash-filter-mutate (3.1.3)
logstash-filter-ruby (3.0.2)
logstash-filter-sleep (3.0.3)
logstash-filter-split (3.1.1)
logstash-filter-syslog_pri (3.0.2)
logstash-filter-throttle (4.0.1)
logstash-filter-urldecode (3.0.3)
logstash-filter-useragent (3.1.1)
logstash-filter-uuid (3.0.2)
logstash-filter-xml (4.0.2)
logstash-input-beats (3.1.18)
logstash-input-couchdb_changes (3.1.1)
logstash-input-elasticsearch (4.0.3)
logstash-input-exec (3.1.2)
logstash-input-file (4.0.0)
logstash-input-ganglia (3.1.0)
logstash-input-gelf (3.0.2)
logstash-input-generator (3.0.2)
logstash-input-graphite (3.0.2)
logstash-input-heartbeat (3.0.2)
logstash-input-http (3.0.4)
logstash-input-http_poller (3.3.0)
logstash-input-imap (3.0.2)
logstash-input-irc (3.0.2)
logstash-input-jdbc (4.2.0)
logstash-input-kafka (5.1.7)
logstash-input-log4j (3.0.5)
logstash-input-lumberjack (3.1.1)
logstash-input-pipe (3.0.2)
logstash-input-rabbitmq (5.2.3)
logstash-input-redis (3.1.2)
logstash-input-s3 (3.1.4)
logstash-input-snmptrap (3.0.2)
logstash-input-sqs (3.0.3)
logstash-input-stdin (3.2.2)
logstash-input-syslog (3.2.0)
logstash-input-tcp (4.1.0)
logstash-input-twitter (3.0.4)
logstash-input-udp (3.1.0)
logstash-input-unix (3.0.3)
logstash-input-xmpp (3.1.2)
logstash-output-cloudwatch (3.0.4)
logstash-output-csv (3.0.3)
logstash-output-elasticsearch (7.3.5)
logstash-output-file (4.0.1)
logstash-output-graphite (3.1.1)
logstash-output-http (4.3.0)
logstash-output-irc (3.0.2)
logstash-output-kafka (5.1.6)
logstash-output-nagios (3.0.2)
logstash-output-null (3.0.2)
logstash-output-pagerduty (3.0.3)
logstash-output-pipe (3.0.2)
logstash-output-rabbitmq (4.0.7)
logstash-output-redis (3.0.3)
logstash-output-s3 (4.0.7)
logstash-output-sns (4.0.3)
logstash-output-sqs (4.0.1)
logstash-output-statsd (3.1.1)
logstash-output-stdout (3.1.0)
logstash-output-tcp (4.0.0)
logstash-output-udp (3.0.2)
logstash-output-webhdfs (3.0.2)
logstash-output-xmpp (3.0.2)
logstash-patterns-core (4.1.0)
x-pack (5.4.3)

As you can see, the dissect filter is clearly included.

My question now is, How did you install Logstash?

[colinsurprenant]

@Paraomao also, could you share with us the content of both these files: Gemfile and Gemfile.jruby-1.9.lock ?

[ThomasADavis]

It is in the rpm's, but it still complains.. something is wrong. logstash.txt

[colinsurprenant]

@ThomasADavis I am not sure this is the same error, it might, but in your case this is the underlying error:

[2017-07-07T11:18:16,031][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Couldn't find any filter plugin named 'dissect'. Are you sure this is correct? Trying to load the dissect filter plugin resulted in this error: Problems loading the requested plugin named dissect of type filter. Error: RuntimeError \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- org/logstash/dissect/jruby-dissect-library/1.0.9/jruby-dissect-library-1.0.9 (LoadError)"}

so it seems /jruby-dissect-library-1.0.9 is missing ?!

[colinsurprenant]

@ThomasADavis could you run this in your logstash home?

find . -name "jruby-dissect-library*" -print

[ThomasADavis]

[root@ls-test logstash]# find . -name "jruby-dissect-library*" -print ./vendor/bundle/jruby/1.9/gems/logstash-filter-dissect-1.0.9/lib/jruby-dissect-library_jars.rb

[ThomasADavis]

I attached an file to an previous comment that shows:

1) install clean logstash 2) simple config 3) start logstash 4) logstash error log 5) rpm contents of logstash with dissect plugin

[colinsurprenant]

@ThomasADavis ok this is very weird, we are investigating with @untergeek.

[colinsurprenant]

I confirm that the dissect jar file is missing from the 5.5.0 RPM package. @ThomasADavis can you try to uninstall and reinstall the dissect filter see if that can be a workaround?

[untergeek]

[root@centos7-pkg-test ~]# /usr/share/logstash/bin/logstash --version
logstash 5.4.0
[root@centos7-pkg-test ~]# /usr/share/logstash/bin/logstash-plugin list --verbose | grep diss
logstash-filter-dissect (1.0.8)

The config:

input { stdin { } }

filter {
  dissect { mapping => { "message" => "%{a}=%{b}" } }
}

output { stdout { codec => rubydebug } }

The output:

[root@centos7-pkg-test ~]# /usr/share/logstash/bin/logstash -f ./test.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs to console
18:56:21.114 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
18:56:21.169 [LogStash::Runner] INFO  logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"9bec4260-48d3-4903-90e0-bbfb821b686b", :path=>"/usr/share/logstash/data/uuid"}
18:56:21.672 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
18:56:21.805 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
18:56:21.938 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
1=2
{
             "a" => "1",
             "b" => "2",
    "@timestamp" => 2017-07-07T18:56:27.677Z,
      "@version" => "1",
          "host" => "centos7-pkg-test.untergeek.net",
       "message" => "1=2"
}

[untergeek]

5.4.3 upgrade...

[root@centos7-pkg-test ~]# rpm -Uvh logstash-5.4.3.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:5.4.3-1               ################################# [ 50%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash
Cleaning up / removing...
   2:logstash-1:5.4.0-1               ################################# [100%]
[root@centos7-pkg-test ~]# /usr/share/logstash/bin/logstash --version
logstash 5.4.3
[root@centos7-pkg-test ~]# /usr/share/logstash/bin/logstash-plugin list --verbose | grep diss
logstash-filter-dissect (1.0.8)
[root@centos7-pkg-test ~]# /usr/share/logstash/bin/logstash -f ./test.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
19:01:07.989 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1000}
19:01:08.131 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
The stdin plugin is now waiting for input:
19:01:08.290 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
a=b
{
             "a" => "a",
             "b" => "b",
    "@timestamp" => 2017-07-07T19:01:12.345Z,
      "@version" => "1",
          "host" => "centos7-pkg-test.untergeek.net",
       "message" => "a=b"
}
^C19:01:54.969 [SIGINT handler] WARN  logstash.runner - SIGINT received. Shutting down the agent.
19:01:54.980 [LogStash::Runner] WARN  logstash.agent - stopping pipeline {:id=>"main"}

[untergeek]

5.5.0 upgrade...

[root@centos7-pkg-test ~]# rpm -Uvh logstash-5.5.0.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:5.5.0-1               warning: /etc/logstash/logstash.yml created as /etc/logstash/logstash.yml.rpmnew
################################# [ 50%]
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash
Cleaning up / removing...
   2:logstash-1:5.4.3-1               ################################# [100%]
[root@centos7-pkg-test ~]# /usr/share/logstash/bin/logstash --version
logstash 5.5.0
[root@centos7-pkg-test ~]# /usr/share/logstash/bin/logstash-plugin list --verbose | grep diss
logstash-filter-dissect (1.0.9)
[root@centos7-pkg-test ~]# /usr/share/logstash/bin/logstash -f ./test.conf
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
19:03:16.573 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
19:03:16.919 [LogStash::Runner] ERROR logstash.plugins.registry - Problems loading a plugin with {:type=>"filter", :name=>"dissect", :path=>"logstash/filters/dissect", :error_message=>"\n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- org/logstash/dissect/jruby-dissect-library/1.0.9/jruby-dissect-library-1.0.9 (LoadError)", :error_class=>RuntimeError, :error_backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/jar-dependencies-0.3.11/lib/jar_dependencies.rb:348:in `do_require'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/jar-dependencies-0.3.11/lib/jar_dependencies.rb:255:in `require_jar'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/jar-dependencies-0.3.11/lib/jar_dependencies.rb:0:in `require_jar_with_block'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/jar-dependencies-0.3.11/lib/jar_dependencies.rb:254:in `require_jar'", "/usr/share/logstash/lib/bootstrap/patches/jar_dependencies.rb:6:in `require_jar'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dissect-1.0.9/lib/jruby-dissect-library_jars.rb:4:in `(root)'", "org/jruby/RubyKernel.java:1040:in `require'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dissect-1.0.9/lib/logstash/filters/dissect.rb:1:in `(root)'", "org/jruby/RubyKernel.java:1040:in `require'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dissect-1.0.9/lib/logstash/filters/dissect.rb:6:in `(root)'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:1:in `(root)'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:156:in `legacy_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:138:in `lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:180:in `lookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:in `lookup'", "org/jruby/RubyKernel.java:1079:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:100:in `plugin'", "(eval):12:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:156:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:314:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
19:03:16.955 [LogStash::Runner] ERROR logstash.agent - Cannot create pipeline {:reason=>"Couldn't find any filter plugin named 'dissect'. Are you sure this is correct? Trying to load the dissect filter plugin resulted in this error: Problems loading the requested plugin named dissect of type filter. Error: RuntimeError \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- org/logstash/dissect/jruby-dissect-library/1.0.9/jruby-dissect-library-1.0.9 (LoadError)"}

[untergeek]

@Paraomao I can confirm a missing jar file in 5.5.0. Whatever you were experiencing in 5.4.x is a different problem.

[colinsurprenant]

Just realized that the missing jar is actually in the logstash-filter-dissect v1.0.9 which bundled in the latest 5.5.0 release.

[ThomasADavis]

[root@ls-test ~]# /usr/share/logstash/bin/logstash-plugin remove logstash-filter-dissect Successfully removed logstash-filter-dissect [root@ls-test ~]# /usr/share/logstash/bin/logstash-plugin install logstash-filter-dissect Validating logstash-filter-dissect Installing logstash-filter-dissect which: no javac in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin) Installation successful [root@ls-test ~]# systemctl restart logstash

still got this:

[2017-07-07T12:21:05,027][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Couldn't find any filter plugin named 'dissect'. Are you sure this is correct? Trying to load th e dissect filter plugin resulted in this error: Problems loading the requested plugin named dissect of type filter. Error: RuntimeError \n\n\tyou might need to reinstall the gem which depend s on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command\n\nno such file to load -- org/logstash/dissect/jruby-dissect-library/1.0.9/jruby-dissect-li brary-1.0.9 (LoadError)"} [2017-07-07T12:21:14,042][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"dissect", :path=>"logstash/filters/dissect", :error_message=>"\n\n\tyou m ight need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command\n\nno such file to load -- org/logstash/dissect/j ruby-dissect-library/1.0.9/jruby-dissect-library-1.0.9 (LoadError)", :error_class=>RuntimeError, :error_backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/jar-dependencies-0.3.11/ lib/jar_dependencies.rb:348:in do_require'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/jar-dependencies-0.3.11/lib/jar_dependencies.rb:255:inrequire_jar'", "/usr/share/logstash/ve ndor/bundle/jruby/1.9/gems/jar-dependencies-0.3.11/lib/jar_dependencies.rb:0:in require_jar_with_block'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/jar-dependencies-0.3.11/lib/jar_d ependencies.rb:254:inrequire_jar'", "/usr/share/logstash/lib/bootstrap/patches/jar_dependencies.rb:6:in require_jar'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-di ssect-1.0.9/lib/jruby-dissect-library_jars.rb:4:in(root)'", "org/jruby/RubyKernel.java:1040:in require'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb: 65:inrequire'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dissect-1.0.9/lib/logstash/filters/dissect.rb:1:in (root)'", "org/jruby/RubyKernel.java:1040:inrequire' ", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dissect-1.0.9/lib/logs tash/filters/dissect.rb:6:in(root)'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:1:in (root)'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.r b:156:inlegacy_lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:138:in lookup'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb:180:inl ookup_pipeline_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:140:in lookup'", "org/jruby/RubyKernel.java:1079:ineval'", "/usr/share/logstash/logstash-core/lib/logsta sh/pipeline.rb:100:in plugin'", "(eval):16:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:72:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash /pipeline.rb:156:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:286:in create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:inregiste r_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:314:in execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/ usr/share/logstash/lib/bootstrap/environment.rb:71:in (root)'"]} [2017-07-07T12:21:14,056][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Couldn't find any filter plugin named 'dissect'. Are you sure this is correct? Trying to load th e dissect filter plugin resulted in this error: Problems loading the requested plugin named dissect of type filter. Error: RuntimeError \n\n\tyou might need to reinstall the gem which depend s on the missing jar or in case there is Jars.lock then resolve the jars withlock_jars` command\n\nno such file to load -- org/logstash/dissect/jruby-dissect-library/1.0.9/jruby-dissect-li brary-1.0.9 (LoadError)"}

[colinsurprenant]

@ThomasADavis ok, yes, normal since the jar file is missing from the 1.0.9 version. can you try uninstalling and installing v1.0.8 instead ?? (so in fact downgrading version)

[ThomasADavis]

[root@ls-test ~]# /usr/share/logstash/bin/logstash-plugin install --version=1.0.8 logstash-filter-dissect Validating logstash-filter-dissect-1.0.8 Installing logstash-filter-dissect which: no javac in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin) Installation successful [root@ls-test ~]# systemctl restart logstash [root@ls-test ~]# more /var/log/logstash/logstash-plain.log [2017-07-07T12:38:45,621][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_infli ght"=>500} [2017-07-07T12:38:45,684][INFO ][logstash.inputs.stdin ] Automatically switching from json to json_lines codec {:plugin=>"stdin"} [2017-07-07T12:38:45,689][INFO ][logstash.pipeline ] Pipeline main started [2017-07-07T12:38:45,729][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2017-07-07T12:38:48,704][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}

[colinsurprenant]

@ThomasADavis great! so we have a workaround in downgrading to v1.0.8 until we release v1.0.10 shortly to provide an upgrade path instead. Thanks for your help with this!

[colinsurprenant]

Also locally confirm the downgrade to v1.0.8 workaround using the tar.gz 5.5.0 package.

[colinsurprenant]

dissect filter version 1.0.11 has been pushed and fixes the issue. From now on a simple plugin update should fix the problem:

bin/logstash-plugin update logstash-filter-dissect

[Paraomao]

Hello, I have installed LS by rpm repo, the dissect plugin is bundled (v 1.0.8). The reported plugin list is short because I have removed all the unneeded plugins, but I got the same error even without removing that plugins.

I have verified the the plugin versions 1.0.8 and 1.0.11 both works using LS 5.4.0 and LS 5.5.0

Note that the dissect v 1.0.9 is bundled in LS 5.5.0, so it does not work out of box.

Thanks!

[colinsurprenant]

Great, thanks @Paraomao for reporting this and confirming the fix. Yes, the fact that the broken 1.0.9 is bundled in 5.5.0 is really unfortunate, we pinned a topic in Discuss for this and will correct this in upcoming 5.5.1

I will go ahead and close this issue.

[qhh0205]

@Paraomao @colinsurprenant I also has the same problem, missing the jar file, my dissect version is v1.1.1(Newest version). I installed locally(git clone-->gem build-->bin/logstash-plugin install /path/to/xxx.gem). So how to solve this problem by myself?

[guyboertje]

@qianghaohao

version 1.1.1 was released to rubygems earlier today.

For reference (file ci/build.sh has clues):

    bundle install
    bundle exec rake vendor
    gem build ./logstash-filter-dissect.gemspec
    gem install ./logstash-filter-dissect-1.1.1.gem