search

opensearch-project / logstash-output-opensearch

A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.
https://opensearch.org/docs/latest/clients/logstash/index/
Apache License 2.0
106 stars 80 forks source link

[BUG]Plugin isn't recognized by Logstash after new version of Logstash is installed #142

Open reshippie opened 2 years ago

reshippie commented 2 years ago

Describe the bug Starting with Logtash 7.16.2 I noticed that the output plugin is no longer recognized if I restart the service after an update to Logstash is installed. I've had to make it a point to reinstall the plugin whenever I restart Logstash to pick up configuration changes or binary updated.

To Reproduce Steps to reproduce the behavior:

  1. Install Logstash >= 7.16.2
  2. Install logstash-output-opensearch
  3. Have Logstash write data to an Opensearch cluster
  4. Install a newer version of Logstash
  5. Restart Logstash.
  6. The output plugin is no longer recognized and Logstash can't write to the Opensearch cluster anymore. 
    [2022-04-26T23:27:11,691][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"output", :name=>"opensearch"}
[2022-04-26T23:27:11,703][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:syslog, :exception=>"
Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'opensearch'. Are you
sure this is correct? Trying to load the opensearch output plugin resulted in this error: Unable to load the requested plugin named opensearch of type output. Th
e plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineEx
t.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initializ
e.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHel
pers.java:1169)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:84)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(I
nterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods
.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jru
by.runtime.callsite.CachingCallSite.call(CachingCallSite.java:85)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInst
ance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logst
ash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minu
s_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.
jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.jav
a:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/us
r/share/logstash/logstash-core/lib/logstash/agent.rb:383)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.
IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.Rub
yProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}

Expected behavior After a restart I would expect that the plugin would still be recognized by the newly updated Logstash package.

Plugins I don't have a list of plugins right now because the command to list them is giving me errors, but the Opensearch output plugin is the only one I have explicitly installed and I haven't explicitly uninstalled any either.

Host/Environment (please complete the following information):

mazhiyuan-aiit commented 2 years ago

I try to install this plugin on logstash version of 7.16.3 is successful， but I try to install this plugin on logstash version of 8.0.0 or 8.2.0 is failure。 I guess it only supports up to logstash version of 7.16.3。

mazhiyuan-aiit commented 2 years ago

I hope the author can continue to support logstash subsequent versions. thank you very much

mazhiyuan-aiit commented 2 years ago

I try to set ECS compability mode to disabled in my configuration , then it is successful Environment: • Image Version: Logstash 8.2.0 • Plugin: 1.2.0

brijos commented 2 years ago

Hi all, I'm tracking the issue. If you have time to help contribute, this should be an easy one.

GlenRSmith commented 2 years ago

Trying to reproduce this, I tried a lesser set of steps - not running Logstash at all.

On Ubuntu 22.04, installed Elastic keys and repos, then

This error does not happen without the plugin installation, and it doesn't happen if I substitute various other plugins, e.g. logstash-input-github.