search

fabric8io / kubernetes-client

Java client for Kubernetes & OpenShift
http://fabric8.io
Apache License 2.0
3.38k stars 1.46k forks source link

Deadlock when there is a wrongly configured client #5824

Closed AndresSanchezG closed 3 weeks ago

AndresSanchezG commented 5 months ago

Describe the bug

Deadlock seems to happen when a jenkins kubernetes agent is trying to be created but it cannot due to some error. In this case having wrongly configured a container.

Fabric8 Kubernetes Client version

6.4.1

Steps to reproduce

Example:

podTemplate(containers: [
    containerTemplate(name: 'maven', image: 'wrong_name:3.0.0', command: 'sleep', args: '99'),
  ]) {
    retry(count: 100, conditions: [kubernetesAgent(), nonresumable()]) {
    node(POD_LABEL) {
        stage('Get a Maven project') {
            container('maven') {
                stage('Hello')

{                     sh 'echo Hello'                 }
            }
        }
    }
    }
}

"wrong_name" image does not exists and then it retries 100 times.

While this pod is being created, when going to manage/configure and pushing save button; configuration is not saved and an error 504 appears in browser.

Performing a thread dump, this thread appears not to finish:

 Handling POST /manage/configSubmit from 10.133.236.30 : https-jsse-nio-8443-exec-10
java.base@11.0.17/jdk.internal.misc.Unsafe.park(Native Method)
java.base@11.0.17/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base@11.0.17/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
java.base@11.0.17/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
java.base@11.0.17/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
java.base@11.0.17/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:162)
io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:185)
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.watch(BaseOperation.java:611)
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.watch(BaseOperation.java:598)
org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper.watchCloud(Reaper.java:221)
org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper.watchClouds(Reaper.java:194)
org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper$ReaperSaveableListener.onChange(Reaper.java:532)
hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:82)
jenkins.model.Jenkins.save(Jenkins.java:3591)
hudson.BulkChange.commit(BulkChange.java:98)
jenkins.model.Jenkins.doConfigSubmit(Jenkins.java:4036)
java.base@11.0.17/java.lang.invoke.LambdaForm$DMH/0x00000008004c6c40.invokeVirtual(LambdaForm$DMH)
java.base@11.0.17/java.lang.invoke.LambdaForm$MH/0x00000008004c3c40.invoke(LambdaForm$MH)
java.base@11.0.17/java.lang.invoke.LambdaForm$MH/0x00000008004c5040.invoke(LambdaForm$MH)
java.base@11.0.17/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
java.base@11.0.17/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
org.kohsuke.stapler.SelectionInterceptedFunction$Adapter.invoke(SelectionInterceptedFunction.java:36)
org.kohsuke.stapler.verb.HttpVerbInterceptor.invoke(HttpVerbInterceptor.java:48)
org.kohsuke.stapler.SelectionInterceptedFunction.bindAndInvoke(SelectionInterceptedFunction.java:26)
org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:836)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
org.kohsuke.stapler.Stapler.service(Stapler.java:248)
javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:163)
org.jenkinsci.plugins.corsfilter.AccessControlsFilter.doFilter(AccessControlsFilter.java:79)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:76)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:118)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:660)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.base@11.0.17/java.lang.Thread.run(Thread.java:829)

And blocks other threads:

"Handling GET /tcpSlaveAgentListener/ from 10.133.236.30 : https-jsse-nio-8443-exec-2 TcpSlaveAgentListener/index.jelly" Id=36 Group=main BLOCKED on hudson.model.Hudson@42d24189 owned by "Handling POST /manage/configSubmit from 10.133.236.30 : https-jsse-nio-8443-exec-10" Id=44
    at jenkins.model.Jenkins.getAgentProtocols(Jenkins.java:1255)
    -  blocked on hudson.model.Hudson@42d24189
    at hudson.TcpSlaveAgentListener.getAgentProtocolNames(TcpSlaveAgentListener.java:163)
    at java.base@11.0.17/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base@11.0.17/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base@11.0.17/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base@11.0.17/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
    at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
    at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
    at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
    at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
    at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
    at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
    at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
    at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:76)
    at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
    at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:259)
    at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
    at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
    at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99)
    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:66)
    at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:55)
    at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)
    at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)
    at org.kohsuke.stapler.Facet.handleIndexRequest(Facet.java:284)
    at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:104)
    at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:32)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
    at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:172)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:163)
    at org.jenkinsci.plugins.corsfilter.AccessControlsFilter.doFilter(AccessControlsFilter.java:79)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
    at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
    at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
    at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:76)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:118)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
    at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:145)
    at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
    at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:660)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    -  locked org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@44b1df47
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base@11.0.17/java.lang.Thread.run(Thread.java:829)

    Number of locked synchronizers = 1
    - org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker@6e02fcd8

Checking the calls it seems to be a problem happening Kubernetes client code in https://github.com/fabric8io/kubernetes-client/blob/v6.8.1/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/BaseOperation.java#L611 introduced by https://github.com/fabric8io/kubernetes-client/commit/0ebd164432cc2e1e6856a49ed82d3edcf3b84564

THis problem started appearing after change in Jenkins core: https://github.com/jenkinsci/jenkins/blame/jenkins-2.414.2/core/src/main/java/jenkins/model/Jenkins.java#L1254 introduced by https://github.com/jenkinsci/jenkins/commit/81bf589596acdd54e3dafb9a7d4cdb66a0029034

Expected behavior

No deadlock should appear.

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

1.24

Environment

Linux

Fabric8 Kubernetes Client Logs

No response

Additional context

No response

shawkins commented 5 months ago

@AndresSanchezG is it a deadlock, or just that one of your theads is holding a lock that is needed by another? A deadlock isn't clear from what you provided.

Also the lock involved "blocked on hudson.model.Hudson@42d24189" isn't coming from fabric8. Since fabric8 can make blocking calls you may want to reconsider how you are holding that lock.

Are you expecting the watch call to fail fast? That may have to do with the number of retries that are allowed - if the misconfiguration is producing an error that seems recoverable, then we may keep trying the operation.

AndresSanchezG commented 5 months ago

@shawkins, so from the threads and traces it seems that while a pod is trying to start up the lock is not released. May be not a deadlock but there is a situation where the lock is never released and therefore it impacts the system. I have already spoken with Jenkins core developers and it seems to be on the part of the plugin.

stale[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!