search

dynatrace-oss / logstash-output-dynatrace

A Logstash output plugin for sending logs to Dynatrace
4 stars 1 forks source link

Logstash 8.x reports the plugin is missing after each update #37

Closed mattBaumBeneva closed 2 months ago

mattBaumBeneva commented 3 months ago

Describe the bug We are running Logstash 8.x installed with YUM (https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#_yum) on Redhat Linux. Each time we update the Logstash package, we end up with the following error messages in the service logs and our pipeline crashes:

[2024-08-22T12:27:17,105][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"output", :name=>"dynatrace"}
[2024-08-22T12:27:17,112][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'dynatrace'. Are you sure this is correct? Trying to load the dynatrace output plugin resulted in this error: Unable to load the requested plugin named dynatrace of type output. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1379)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:949)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:354)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:1583)"]}

When this occurs, we must reinstall the plugin to fix the issue.

To Reproduce Steps to reproduce the behavior:

  1. Install a previous version of Logstash 8.x with yum on Redhat 8
  2. Install the Dynatrace output plugin
  3. Update the logstash package to the latest version
  4. Restart the logstash systemd service
  5. View logs with: journalctl -u logstash
  6. See error

Expected behavior Upgrading logstash should have no effect on previously-installed output plugins

Additional context It isn't clear if this is a pluging-related issue of if this is a bug in the logstash update process. We see the bug affecting the dynatrace output plugin when we update.

mattBaumBeneva commented 3 months ago

From what we can tell, the file still exists under /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems, but logstash does not appear to load it correctly.

mattBaumBeneva commented 3 months ago

We have also opened a bug report with the Logstash project here, in case this is a logstash issue: https://github.com/elastic/logstash/issues/16400

dyladan commented 3 months ago

I agree updating logstash should not affect installed plugins. Can you please answer a couple questions for me:

  1. Is this also affecting other plugins, or just Dynatrace?
  2. Are you observing this only in the case that you update logstash from a previous version, or does it also happen if you start with the latest logstash and install the plugin?
  3. What exact steps did you take to install logstash? Was this done as root?
  4. What exact steps did you take to upgrade logstash? Was this done as root?
  5. What exact steps did you take to install the plugin? Was this done as root?

From what we can tell, the file still exists under /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems, but logstash does not appear to load it correctly.

My best theory right now is that the upgrade caused a permissions issue. I've seen cases in the past where the plugin is in the correct directory, but the user the logstash process runs as does not have sufficient permission to load it. This could happen if the upgrade caused the user logstash runs as to change, or if yum "fixed" what it viewed as incorrect permissions in the plugin directory.

mattBaumBeneva commented 3 months ago
  1. We may have cases on other VMs where the behavior affects other plugins, but this remains to be confirmed
  2. When initially installing, it works
  3. The following steps were performed (through sudo):
  1. Our company runs "yum update" on all VMs regularly using Ansible Automation Platform. Ansible runs under a dedicated user (probably runs the command a root)
  2. Like so:

You are correct about the permissions issue. Following an update, the permissions on /usr/share/logstash/* revert to root. We tried applying ACLs before updating: setfacl -Rm d:u:logstash:rwx /usr/share/logstash setfacl -Rm u:logstash:rwx /usr/share/logstash

Which avoids permission errors, but the error concerning the plugin not being found persists.

dyladan commented 3 months ago

We tried applying ACLs before updating: setfacl -Rm d:u:logstash:rwx /usr/share/logstash setfacl -Rm u:logstash:rwx /usr/share/logstash

Which avoids permission errors, but the error concerning the plugin not being found persists.

Did you mean to say after updating? The plugin is still not loaded even with correct permissions?

Have you verified that installing the plugin on a scratch installation of the latest target version of logstash works, rather than updating from a previous version? If so, was the plugin installed in the same directory? I'm wondering if the location logstash-plugin installs to might have changed? According to the release notes the bundled version of JRuby was updated along with the bundled JVM. The new JRuby still targets ruby 3.1 so I would expect the installation directory to stay the same. My gut feeling is that this is still somehow related to permissions.

A second possibility is that a Gemfile (or some other configuration file) in /usr/share/logstash was modified during the Yum update and caused the issue.

dyladan commented 3 months ago

I'll set up a VM this afternoon so I can do some investigation as well

mattBaumBeneva commented 3 months ago

I mean that we applied the ACLs before running yum update to grant RWX access to the logstash user, even if the update changes ownership to "root". In that case, we didn't have any permissions errors in the service logs, but logstash still complained that the plugin no longer existed.

We have not tested a fresh install in a while. We set up these logstash instances about 3-4 months ago and they have been runnign into this issues each month since (when the automatic update process runs). We haven't tested this from scratch on a VM without Logstash since that time.

My gut feeling is that a permission issue interferes with the upgrade process and leaves Logstash in an invalid state (the plugin being present but unusable.

Thanks for testing on your end.

dyladan commented 3 months ago

I mean that we applied the ACLs before running yum update to grant RWX access to the logstash user, even if the update changes ownership to "root"

Would this be preserved even if the file was deleted and recreated?

mattBaumBeneva commented 3 months ago

I mean that we applied the ACLs before running yum update to grant RWX access to the logstash user, even if the update changes ownership to "root"

Would this be preserved even if the file was deleted and recreated?

Yes. This is the state of the directory (the files inside have the same privileges) after setting the ACL and running yum update:

image

The directory itself has the following permissions following an update: drwxrwxr-x+ 4 root root 180 Apr 15 14:50 logstash-output-dynatrace-0.6.0

Output of "yum info logstash":

Name         : logstash
Epoch        : 1
Version      : 8.15.0
Release      : 1
Architecture : x86_64
Size         : 665 M
Source       : logstash-8.15.0-1.src.rpm
Repository   : @System
From repo    : logstash-8.x
Summary      : An extensible logging pipeline
URL          : https://www.elastic.co/logstash
License      : Elastic License
Description  : An extensible logging pipeline
mattBaumBeneva commented 2 months ago

We have been able to confirm this issue also occurs with the syslog output plugin. Have you had any success reproducing?

dyladan commented 2 months ago

Yes I was able to reproduce it. I still think the problem lies in permissions/ownership. In my experience, you should not have to manually change permissions, or especially ownership, for applications installed through package managers. I'm able to get logstash to work properly if I re-run the "change ownership" step after the yum update.

I am also able to get logstash to work properly, including updating, if I never change ownership of the /usr/share/logstash directory or its contents, which leads me to believe this is the root of the issue.

mattBaumBeneva commented 2 months ago

Thanks, @dyladan ! I agree this is the root cause. I assume this is more of a Logstash bug than a plugin bug. Do you think its possible to get this prioritized and investigated on the Logstash end of things? We opened the same issue with the Elastic/Logstash project, but no movement since, and this is having an important impact on our systems. Can we put some weight on this request through our Dynatrace support contract?

dyladan commented 2 months ago

I'm sorry I don't have any influence over the logstash project. It seems most likely to be an issue with the way logstash and yum interact. Is there some reason you don't want to just let the logstash directory be owned by root the way yum sets it up?

Any questions you have about support should go through your support rep.

mattBaumBeneva commented 2 months ago

We'll follow up with our support rep. Yum grants root privileges but runs the service under a logstash group and user, not root (which is the more secure way to run services). Thanks for confirming our analysis of the issue. We now understand it isn't specific to the Dynatrace plugin.

De : Daniel Dyla @.> Envoyé : vendredi 20 septembre 2024 11:32 À : dynatrace-oss/logstash-output-dynatrace @.> Cc : Baum, Matthew @.>; Author @.> Objet : Re: [dynatrace-oss/logstash-output-dynatrace] Logstash 8.x reports the plugin is missing after each update (Issue #37)

I'm sorry I don't have any influence over the logstash project. It seems most likely to be an issue with the way logstash and yum interact. Is there some reason you don't want to just let the logstash directory be owned by root the way yum sets it up?

Any questions you have about support should go through your support rep.

— Reply to this email directly, view it on GitHubhttps://github.com/dynatrace-oss/logstash-output-dynatrace/issues/37#issuecomment-2363998809, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A6MEM3EWP42NFWLQZ3ZADMLZXQ5Z3AVCNFSM6AAAAABM6R36RWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRTHE4TQOBQHE. You are receiving this because you authored the thread.

AVIS: Ce courriel privilégié et confidentiel est destiné à la seule personne ou entité à laquelle il est adressé. Pour toute autre personne, toute action prise en rapport à ce courriel ainsi que toute lecture, reproduction, transmission et/ou divulgation d'une partie ou de l'ensemble de celui-ci est interdite. Si vous n'êtes pas la personne autorisée à recevoir ce courriel, S.V.P. le retourner à l'expéditeur et le détruire. Bien que ce courriel ait été traité contre les virus, il est de la responsabilité du destinataire de s'assurer que l'envoi en est exempt. Nos communications avec vous peuvent contenir des renseignements confidentiels ou protégés par le secret professionnel. Si vous désirez que nous communiquions avec vous par un autre moyen de transmission que le courrier électronique ordinaire non sécurisé, veuillez nous en aviser.

NOTICE: This privileged and confidential email is intended only for the individual or entity to whom it is addressed. With regard to all others, any action related with this email as well as any reading, reproduction, transmission and/or dissemination in whole or in part of the information included in this email is prohibited. If you are not the addressee, immediately return the email to sender prior to destroying all copies. Even if this email is believed to be free from any virus, it is the responsibility of the recipient to make sure that it is virus exempt. Our communications to you may contain confidential information or information protected under solicitor-client privilege. Please advise if you wish us to use a mode of communication other than regular, unsecured e-mail in our communications with you.