I've noticed a deadlock when calling the GerritConnection.shutdown(true) method while establishing the connection in GerritConnection.run().
The deadlock sequence is like this:
Thread 1: calls GerritConnection.run() -> calls into connect() -> SshConnectionFactory.getConnection()
Thread 2: calls GerritConnection.shutdown(true), sshConnection is null at this point, doesn't call sshConnection.disconnect(), calls join()
Thread 1: returns from connect() call, sets sshConnection, continues up to read() and blocks
Thread 2: join() never returns as read() in Thread 1 blocks (sshConnection.disconnect() is not called before)
Below are the callstacks of the two threads (Thread 2 initiates the connection shutdown by executing a Jenkins script "io.jenkins.plugins.casc.ConfigurationAsCode.get().configure()", JCasC removes all Gerrit severs and adds them again).
Id=106 Group=main TIMED_WAITING on com.jcraft.jsch.Channel$MyPipedInputStream@686f0414
at java.base@11.0.21/java.lang.Object.wait(Native Method)
- waiting on com.jcraft.jsch.Channel$MyPipedInputStream@686f0414
at java.base@11.0.21/java.io.PipedInputStream.read(PipedInputStream.java:326)
at java.base@11.0.21/java.io.PipedInputStream.read(PipedInputStream.java:377)
at java.base@11.0.21/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base@11.0.21/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base@11.0.21/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked java.io.InputStreamReader@450eb826
at java.base@11.0.21/java.io.InputStreamReader.read(InputStreamReader.java:181)
at java.base@11.0.21/java.io.Reader.read(Reader.java:189)
at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.run(GerritConnection.java:434)
"Handling POST /scriptText from 172.17.0.4 : Jetty (winstone)-35" Id=35 Group=main WAITING on com.sonymobile.tools.gerrit.gerritevents.GerritConnection@28a5ee8e
at java.base@11.0.21/java.lang.Object.wait(Native Method)
- waiting on com.sonymobile.tools.gerrit.gerritevents.GerritConnection@28a5ee8e
at java.base@11.0.21/java.lang.Thread.join(Thread.java:1300)
at java.base@11.0.21/java.lang.Thread.join(Thread.java:1375)
at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:710)
at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:617)
- locked com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer@29949ddb
at com.sonyericsson.hudson.plugins.gerrit.trigger.config.GerritJcascConfigurator.configure(GerritJcascConfigurator.java:95)
at com.sonyericsson.hudson.plugins.gerrit.trigger.config.GerritJcascConfigurator.configure(GerritJcascConfigurator.java:57)
at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:277)
at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:360)
at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:283)
at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$configureWith$7(ConfigurationAsCode.java:823)
at io.jenkins.plugins.casc.ConfigurationAsCode$$Lambda$561/0x0000000800ca8c40.apply(Unknown Source)
at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:773)
at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:823)
at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:695)
at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:352)
at io.jenkins.plugins.casc.ConfigurationAsCode$configure$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
at Script1.run(Script1.groovy:1)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:574)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:612)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:583)
at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:149)
at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:118)
at hudson.remoting.LocalChannel.call(LocalChannel.java:47)
at hudson.util.RemotingDiagnostics.executeGroovy(RemotingDiagnostics.java:115)
at jenkins.model.Jenkins._doScript(Jenkins.java:4869)
at jenkins.model.Jenkins.doScriptText(Jenkins.java:4847)
at java.base@11.0.21/java.lang.invoke.LambdaForm$DMH/0x00000008001ff440.invokeVirtual(LambdaForm$DMH)
at java.base@11.0.21/java.lang.invoke.LambdaForm$MH/0x0000000800422040.invoke(LambdaForm$MH)
at java.base@11.0.21/java.lang.invoke.LambdaForm$MH/0x00000008003f1440.invoke(LambdaForm$MH)
at java.base@11.0.21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
at java.base@11.0.21/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:163)
at hudson.security.HudsonPrivateSecurityRealm$2.doFilter(HudsonPrivateSecurityRealm.java:1022)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:39)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:76)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at jenkins.ErrorAttributeFilter.doFilter(ErrorAttributeFilter.java:29)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at jenkins.security.ApiCrumbExclusion.process(ApiCrumbExclusion.java:48)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:128)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:139)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:86)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:569)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
at java.base@11.0.21/java.lang.Thread.run(Thread.java:829)
I've noticed a deadlock when calling the
GerritConnection.shutdown(true)
method while establishing the connection in GerritConnection.run().The deadlock sequence is like this:
Thread 1: calls GerritConnection.run() -> calls into connect() -> SshConnectionFactory.getConnection() Thread 2: calls GerritConnection.shutdown(true), sshConnection is null at this point, doesn't call sshConnection.disconnect(), calls join() Thread 1: returns from connect() call, sets sshConnection, continues up to read() and blocks Thread 2: join() never returns as read() in Thread 1 blocks (sshConnection.disconnect() is not called before)
Below are the callstacks of the two threads (Thread 2 initiates the connection shutdown by executing a Jenkins script "io.jenkins.plugins.casc.ConfigurationAsCode.get().configure()", JCasC removes all Gerrit severs and adds them again).