"Issue with 'dynatrace' Output Plugin Configuration: Unable to configure plugins"

[mdAliMaaz]

 Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'dynatrace'. Are you sure this is correct? Trying to load the dynatrace output plugin resulted in this error: Unable to load the requested plugin named dynatrace of type output. The plugin is not installed."

Environment:

Operating System: Rhel 8.4

Steps to Reproduce:

1. install logstash
2. install logstash-output-dynatrace
3. add configuration file.
4. start logstash
5. observe error message in the Logstash Logs

Expected Behavior:

The 'dynatrace' output plugin should be loaded successfully, and Logstash should start without errors.

Actual Behavior:

Logstash fails to start due to an error loading the 'dynatrace' output plugin. The error message indicates that the plugin cannot be found or is not installed correctly.

Additional Information:

our configuration file looks like this

input {   file

        {
         path => <PATH_TO_LOGS>
         start_position => "beginning"
         sincedb_path => "/dev/null"
        }
      }
filter { }
output {
        #stdout { codec => json }
        dynatrace {
                ingest_endpoint_url => <URL>
                api_key => <API_TOKEN>
                ssl_verify_none => true
        }
}

actual logs

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'dynatrace'. Are you sure this is correct? Trying to load the dynatrace output plugin resulted in this error: Unable to load the requested plugin named dynatrace of type output. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1319)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:840)"]}

• we have Ensured that the 'dynatrace' output plugin is installed correctly on the Logstash instance.
• Double-check the plugin configuration in the Logstash configuration file to ensure it is spelled correctly and includes all required parameters.

[dyladan]

Can you provide the exact steps you used to install the plugin and the exact steps you used to check it is installed correctly? What versions of logstash and the output plugin are installed?

What user is logstash running as, and what user did you use to install the plugin? It is a somewhat common issue that a plugin is installed as root and the logstash user which the service runs as doesn't have correct permissions to load it.

[mdAliMaaz]

Download and install public key

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo

[logstash-8.x]
name=Elastic repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

sudo yum install logstash

Logstash is typically installed in the /usr/share/logstash directory, and plugins are installed using the /usr/share/logstash/bin/logstash-plugin command. If your logstash installation directory is different than this, your logstash-plugin command may be in a different location.

/usr/share/logstash/bin/logstash-plugin install logstash-output-dynatrace

Now lets do configuartion

 vim /etc/logstash/conf.d/dtingest.conf

and past this code

input {   file
        {
         path => "<PATH_TO_LOGS>"
         start_position => "beginning"
         sincedb_path => "/dev/null"
        }
      }
filter { }
output {
        #stdout { codec => json }
        dynatrace {
                ingest_endpoint_url => "<YOUR_API_URL>"
                api_key => "<YOUR_API_TOKEN>"
                ssl_verify_none => true
        }
}

Now let's run it by running this command.

 ./logstash -f /etc/logstash/conf.d/dtingest.conf 

Everything is installed and running with root user permission. After executing the above command we can't see any error or output. Thank you.

[dyladan]

we can't see any error or output

No error is good. Are you not seeing the Failed to execute action from before?

or output

Does this mean you're not seeing logs in Dynatrace or that you're not seeing console logs from logstash?

[mdAliMaaz]

When we are running Logstash, sometimes it doesn't work, and even when it does, it's really slow in processing logs (meaning we're able to see logs in the web UI after a very long delay).

[dyladan]

Is there any update on this? If you're not seeing any errors I'm not sure what would be causing logs to be slow to arrive. It seems to me that your logstash installation may be corrupt in some way. This is often caused by permissions issues related to installing or running as root.