search

sonyxperiadev / gerrit-events

MIT License
47 stars 62 forks source link

Deadlock when shutting down while establishing connection #126

Open ckullabosch opened 7 months ago

ckullabosch commented 7 months ago

I've noticed a deadlock when calling the GerritConnection.shutdown(true) method while establishing the connection in GerritConnection.run().

The deadlock sequence is like this:

Thread 1: calls GerritConnection.run() -> calls into connect() -> SshConnectionFactory.getConnection() Thread 2: calls GerritConnection.shutdown(true), sshConnection is null at this point, doesn't call sshConnection.disconnect(), calls join() Thread 1: returns from connect() call, sets sshConnection, continues up to read() and blocks Thread 2: join() never returns as read() in Thread 1 blocks (sshConnection.disconnect() is not called before)

Below are the callstacks of the two threads (Thread 2 initiates the connection shutdown by executing a Jenkins script "io.jenkins.plugins.casc.ConfigurationAsCode.get().configure()", JCasC removes all Gerrit severs and adds them again).

Id=106 Group=main TIMED_WAITING on com.jcraft.jsch.Channel$MyPipedInputStream@686f0414
    at java.base@11.0.21/java.lang.Object.wait(Native Method)
    -  waiting on com.jcraft.jsch.Channel$MyPipedInputStream@686f0414
    at java.base@11.0.21/java.io.PipedInputStream.read(PipedInputStream.java:326)
    at java.base@11.0.21/java.io.PipedInputStream.read(PipedInputStream.java:377)
    at java.base@11.0.21/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at java.base@11.0.21/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at java.base@11.0.21/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    -  locked java.io.InputStreamReader@450eb826
    at java.base@11.0.21/java.io.InputStreamReader.read(InputStreamReader.java:181)
    at java.base@11.0.21/java.io.Reader.read(Reader.java:189)
    at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.run(GerritConnection.java:434)

"Handling POST /scriptText from 172.17.0.4 : Jetty (winstone)-35" Id=35 Group=main WAITING on com.sonymobile.tools.gerrit.gerritevents.GerritConnection@28a5ee8e
    at java.base@11.0.21/java.lang.Object.wait(Native Method)
    -  waiting on com.sonymobile.tools.gerrit.gerritevents.GerritConnection@28a5ee8e
    at java.base@11.0.21/java.lang.Thread.join(Thread.java:1300)
    at java.base@11.0.21/java.lang.Thread.join(Thread.java:1375)
    at com.sonymobile.tools.gerrit.gerritevents.GerritConnection.shutdown(GerritConnection.java:710)
    at com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer.stopConnection(GerritServer.java:617)
    -  locked com.sonyericsson.hudson.plugins.gerrit.trigger.GerritServer@29949ddb
    at com.sonyericsson.hudson.plugins.gerrit.trigger.config.GerritJcascConfigurator.configure(GerritJcascConfigurator.java:95)
    at com.sonyericsson.hudson.plugins.gerrit.trigger.config.GerritJcascConfigurator.configure(GerritJcascConfigurator.java:57)
    at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:277)
    at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:360)
    at io.jenkins.plugins.casc.BaseConfigurator.configure(BaseConfigurator.java:283)
    at io.jenkins.plugins.casc.ConfigurationAsCode.lambda$configureWith$7(ConfigurationAsCode.java:823)
    at io.jenkins.plugins.casc.ConfigurationAsCode$$Lambda$561/0x0000000800ca8c40.apply(Unknown Source)
    at io.jenkins.plugins.casc.ConfigurationAsCode.invokeWith(ConfigurationAsCode.java:773)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:823)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:695)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:352)
    at io.jenkins.plugins.casc.ConfigurationAsCode$configure$0.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
    at Script1.run(Script1.groovy:1)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:574)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:612)
    at groovy.lang.GroovyShell.evaluate(GroovyShell.java:583)
    at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:149)
    at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:118)
    at hudson.remoting.LocalChannel.call(LocalChannel.java:47)
    at hudson.util.RemotingDiagnostics.executeGroovy(RemotingDiagnostics.java:115)
    at jenkins.model.Jenkins._doScript(Jenkins.java:4869)
    at jenkins.model.Jenkins.doScriptText(Jenkins.java:4847)
    at java.base@11.0.21/java.lang.invoke.LambdaForm$DMH/0x00000008001ff440.invokeVirtual(LambdaForm$DMH)
    at java.base@11.0.21/java.lang.invoke.LambdaForm$MH/0x0000000800422040.invoke(LambdaForm$MH)
    at java.base@11.0.21/java.lang.invoke.LambdaForm$MH/0x00000008003f1440.invoke(LambdaForm$MH)
    at java.base@11.0.21/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
    at java.base@11.0.21/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
    at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:163)
    at hudson.security.HudsonPrivateSecurityRealm$2.doFilter(HudsonPrivateSecurityRealm.java:1022)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:39)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:76)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at jenkins.ErrorAttributeFilter.doFilter(ErrorAttributeFilter.java:29)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at jenkins.security.ApiCrumbExclusion.process(ApiCrumbExclusion.java:48)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:128)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
    at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:139)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:86)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:569)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
    at org.eclipse.jetty.server.Server.handle(Server.java:563)
    at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.base@11.0.21/java.lang.Thread.run(Thread.java:829)