search

jenkinsci / opentelemetry-plugin

Monitor and observe Jenkins with OpenTelemetry.
https://plugins.jenkins.io/opentelemetry/
Apache License 2.0
95 stars 49 forks source link

Null pointer exception crashing jenkins #678

Closed liron58 closed 11 months ago

liron58 commented 1 year ago

Jenkins and plugins versions report

Environment Jenkins: 2.401.3 OS: Linux - 3.10.0-1160.80.1.el7.x86_64 Java: 13.0.2 - Oracle Corporation (Java HotSpot(TM) 64-Bit Server VM) --- okhttp-api:4.11.0-145.vcb_8de402ef81 opentelemetry:2.14.0

What Operating System are you using (both controller, and any agents involved in the problem)?

CentOS 7

Reproduction steps

I wish I could explain how it happend, after week or more of the plugin working the jenkins have crashed with those errors in every build :

ERROR: Processing failed due to a bug in the code. Please report this to the issue tracker (https://jenkins.io/redirect/report-an-issue).

20:24:48 java.lang.NullPointerException
20:24:48    at io.jenkins.plugins.opentelemetry.job.MonitoringBuildStepListener.started(MonitoringBuildStepListener.java:55)
20:24:48    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:812)
20:24:48    at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.build(MavenModuleSetBuild.java:944)
20:24:48    at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:894)
20:24:48    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:526)
20:24:48    at hudson.model.Run.execute(Run.java:1900)
20:24:48    at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543)
20:24:48    at hudson.model.ResourceController.execute(ResourceController.java:101)
20:24:48    at hudson.model.Executor.run(Executor.java:442)

and this on the jenkins log : 
Caught unhandled exception with ID 68914331-eb64-432a-a28c-efbbe952f93b
java.lang.NoSuchMethodError: 'io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.registerShutdownHook(boolean)'
    at io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider.initializeOtlp(OpenTelemetrySdkProvider.java:158)
    at io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider.initialize(OpenTelemetrySdkProvider.java:131)
    at io.jenkins.plugins.opentelemetry.JenkinsOpenTelemetryPluginConfiguration.initializeOpenTelemetry(JenkinsOpenTelemetryPluginConfiguration.java:222)
    at io.jenkins.plugins.opentelemetry.JenkinsOpenTelemetryPluginConfiguration.configure(JenkinsOpenTelemetryPluginConfiguration.java:165)
    at jenkins.model.Jenkins.configureDescriptor(Jenkins.java:4039)
    at jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:4003)
    at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:719)
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
    at org.kohsuke.stapler.SelectionInterceptedFunction$Adapter.invoke(SelectionInterceptedFunction.java:36)
    at org.kohsuke.stapler.verb.HttpVerbInterceptor.invoke(HttpVerbInterceptor.java:48)
    at org.kohsuke.stapler.SelectionInterceptedFunction.bindAndInvoke(SelectionInterceptedFunction.java:26)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
    at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
Caused: javax.servlet.ServletException
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:818)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:836)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
    at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
    at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
    at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
    at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:154)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
    at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:608)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:764)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1388)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:830)

Expected Results

The plugin will work seemingly and even if failed from some reason dont kill all jenkins jobs

Actual Results

all jenkins job are down

Anything else?

No response

liron58 commented 1 year ago

maybe related : https://github.com/open-telemetry/opentelemetry-java/pull/5565

cyrille-leclerc commented 1 year ago

Great catch, could you have loaded a more recent version of the OtelSDK jar in Jenkins?

cyrille-leclerc commented 11 months ago

Can't reproduce. We have bumped to OTel SDK 1.29.0 and will soon bump to 1.30 which is the version of the latest OTel Java Auto Instrumentation. This should lower the risk of conflicts when mixing OTel Java Auto Instr with the Jenkins OTel Plugin.

Please reopen if you have more details.