search

GoogleCloudPlatform / continuous-deployment-on-kubernetes

Get up and running with Jenkins on Google Kubernetes Engine
Apache License 2.0
1.6k stars 890 forks source link

Something wrong between Master and slave ClosedChannelException #118

Closed KJlmfe closed 6 years ago

KJlmfe commented 6 years ago

I follow this tutorial https://cloud.google.com/solutions/configuring-jenkins-kubernetes-engine

But it doesn't work for me.

Started by timer
Building remotely on jenkins-k8s-slave-psvfp (jenkins-k8s-slave) in workspace /root/workspace/dd
FATAL: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.12.0.38/10.12.0.38:49144
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737)
        at hudson.remoting.Request.call(Request.java:197)
        at hudson.remoting.Channel.call(Channel.java:951)
        at hudson.FilePath.act(FilePath.java:998)
        at hudson.FilePath.act(FilePath.java:987)
        at hudson.FilePath.mkdirs(FilePath.java:1170)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1201)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
        at hudson.model.Run.execute(Run.java:1727)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.remoting.RequestAbortedException
    at hudson.remoting.Request.abort(Request.java:335)
    at hudson.remoting.Channel.terminate(Channel.java:1034)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
    at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:313)
    at hudson.remoting.Channel.close(Channel.java:1446)
    at hudson.remoting.Channel.close(Channel.java:1399)
    at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:746)
    at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:99)
    at hudson.slaves.SlaveComputer$2.onClosed(SlaveComputer.java:534)
    at hudson.remoting.Channel.terminate(Channel.java:1055)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:208)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
    at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:313)
    at hudson.remoting.Channel.close(Channel.java:1446)
    at hudson.remoting.Channel.close(Channel.java:1399)
    at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:604)
    at jenkins.slaves.DefaultJnlpSlaveReceiver.afterChannel(DefaultJnlpSlaveReceiver.java:168)
    at org.jenkinsci.remoting.engine.JnlpConnectionState$4.invoke(JnlpConnectionState.java:421)
    at org.jenkinsci.remoting.engine.JnlpConnectionState.fire(JnlpConnectionState.java:312)
    at org.jenkinsci.remoting.engine.JnlpConnectionState.fireAfterChannel(JnlpConnectionState.java:418)
    at org.jenkinsci.remoting.engine.JnlpProtocol4Handler$Handler$1.run(JnlpProtocol4Handler.java:334)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
viglesiasce commented 6 years ago

This should now be fixed as of the latest updates to the tutorial in #96.

alan-ma-umg commented 6 years ago

Hi Vic,

I just used the most current branch to build a Jenkins in k8s cluster. I have got the following failure, which seems still an issue as originally reported. Could you please advise? Thanks!

FATAL: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.250.17.21/10.250.17.21:56626
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
        at hudson.remoting.Request.call(Request.java:202)
        at hudson.remoting.Channel.call(Channel.java:954)
        at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
        at com.sun.proxy.$Proxy101.isAlive(Unknown Source)
        at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1137)
        at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:166)
        at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:879)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
        at hudson.model.Run.execute(Run.java:1798)
        at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.remoting.RequestAbortedException
    at hudson.remoting.Request.abort(Request.java:340)
    at hudson.remoting.Channel.terminate(Channel.java:1038)
    at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
    at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
    at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
    at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
    at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
    at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
    at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142)
    at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
viglesiasce commented 6 years ago

@alan-ma-umg can you post your pipeline config? Did you change any additional configuration?

viglesiasce commented 6 years ago

Also do you have any additional network policies in place?

alan-ma-umg commented 6 years ago

Hi Vic,

Thanks for following up. I have resolved the issue by setting higher CPU/Memory. The settings locates in manage jenkins->configure system->images->kubernetes pod template->container template->advanced section. HTH for future users.

james-powis commented 6 years ago

As @alan-ma-umg initially suggested, I too had with the slave causing failures on a simple hello world job. Settings the CPU Request and Limit to 400m and Memory to 512mb resolved the issue. I will say these values better align with my use case, so others might want to toy with these values.

richyla commented 6 years ago

Increasing the slave resource requests & limits also solved this issue for me. Thanks.

viglesiasce commented 6 years ago

Updated the defaults in: https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes/pull/133

ghost commented 5 years ago

This thread was a lifesaver. We setup our own pipeline using jenkins on kubernetes and our slave kept crashing due to the same reason. We were using the kubernetes jenkins plugin and the default limits for the pods were not set. On setting the limits, the jobs run just fine!

chethan39 commented 5 years ago

Hi all, I have a similar issue in Jenkins (both Master & Agents on Windows 2102). I don't see the option to increase CPU/ Memory under manage jenkins->configure system. Though i increased memory in jenkins.xml or jenkins-slave.xml it did not help resolve the issue. Can you please guide me with possible solution ?