search

fabric8io / fabric8-pipeline-library

Fabric8 Pipeline for Jenkins
Apache License 2.0
438 stars 211 forks source link

error starting build pod java.io.IOException: Pipe not connected #206

Open rawlingsj opened 7 years ago

rawlingsj commented 7 years ago

Seems to happen when there's low resources or multiple jobs running. This has been seen on OSO and GKE.

Executing shell script inside container [maven] of pod [kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e]
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

java.io.IOException: Pipe not connected
    at java.io.PipedOutputStream.write(PipedOutputStream.java:140)
    at java.io.OutputStream.write(OutputStream.java:75)
    at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:125)
    at hudson.Launcher$ProcStarter.start(Launcher.java:384)
    at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:157)
    at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:63)
    at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:172)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
    at groovy.lang.GroovyObject$invokeMethod.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:151)
    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:21)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:115)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:123)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:16)
    at stageProject.call(/var/jenkins_home/jobs/fabric8-cd/jobs/fabric8-maven-plugin/branches/master/builds/16/libs/github.com/fabric8io/fabric8-pipeline-library/vars/stageProject.groovy:18)
    at ___cps.transform___(Native Method)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
    at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
    at com.cloudbees.groovy.cps.Next.step(Next.java:74)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Finished: FAILURE
rawlingsj commented 7 years ago

this is a dump of the recent jenkins server logs

Excess workload after pending Spot instances: 1
Apr 18, 2017 2:30:29 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Apr 18, 2017 2:30:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Created Pod: kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e
Apr 18, 2017 2:30:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Waiting for Pod to be scheduled (0/100): kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e
Apr 18, 2017 2:30:31 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
Accepted connection #140 from /10.16.3.16:54268
Apr 18, 2017 2:30:39 PM INFO hudson.slaves.NodeProvisioner$2 run
Kubernetes Pod Template provisioning successfully completed. We have now 4 computer(s)
Apr 18, 2017 2:33:12 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.45 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:12 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:12 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:12 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.40 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:12 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.40 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:33:29 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Apr 18, 2017 2:33:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Created Pod: kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6
Apr 18, 2017 2:33:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Waiting for Pod to be scheduled (0/100): kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6
Apr 18, 2017 2:33:30 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
Accepted connection #141 from /10.16.2.192:54688
Apr 18, 2017 2:33:33 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:33 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:33 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:33 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.45 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:33 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.45 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:36 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:36 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:36 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.40 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:36 PM INFO org.jenkinsci.plugins.github_branch_source.PushGHEventSubscriber onEvent
Received PUSH for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.40 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:39 PM INFO hudson.slaves.NodeProvisioner$2 run
Kubernetes Pod Template provisioning successfully completed. We have now 5 computer(s)
Apr 18, 2017 2:33:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:33:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:33:55 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:55 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:56 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:56 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.41 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:56 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.42 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:56 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.41 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:56 PM INFO org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber onEvent
Received PULL_REQUEST for https://github.com/fabric8-ui/fabric8-runtime-console from 192.30.252.45 ⇒ http://jenkins.cd.k8s.fabric8.io/github-webhook/
Apr 18, 2017 2:33:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:33:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:39 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:39 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:34:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:34:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:29 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:39 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:39 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:42 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminating Kubernetes instance for slave kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e
Apr 18, 2017 2:35:42 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminated Kubernetes instance for slave kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e
Apr 18, 2017 2:35:42 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Disconnected computer kubernetes-137ebf2065f949d4acac4e019ed07af7-1e96524904d1e
Apr 18, 2017 2:35:42 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
fabric8-cd/fabric8-maven-plugin/master #16 completed: FAILURE
Apr 18, 2017 2:35:42 PM WARNING hudson.model.listeners.RunListener report
RunListener failed
java.util.ConcurrentModificationException
    at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
    at java.util.TreeMap$KeyIterator.next(TreeMap.java:1265)
    at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.createBuild(BuildListener.java:71)
    at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.onCompleted(BuildListener.java:49)
    at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:211)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:618)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:125)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:901)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1092)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:400)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Apr 18, 2017 2:35:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:49 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:35:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:35:59 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:36:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:36:09 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud addProvisionedSlave
Total container cap of 4 reached, not provisioning: 4 running in namespace cd
Apr 18, 2017 2:36:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
Excess workload after pending Spot instances: 1
Apr 18, 2017 2:36:19 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
Started provisioning Kubernetes Pod Template from kubernetes with 1 executors. Remaining excess workload: 0
Apr 18, 2017 2:36:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Created Pod: kubernetes-e9706a5ff8074c468e1a35dc3516c250-1e9b6a1af6191
Apr 18, 2017 2:36:19 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback call
Waiting for Pod to be scheduled (0/100): kubernetes-e9706a5ff8074c468e1a35dc3516c250-1e9b6a1af6191
Apr 18, 2017 2:36:20 PM INFO hudson.TcpSlaveAgentListener$ConnectionHandler run
Accepted connection #142 from /10.16.3.17:34658
Apr 18, 2017 2:36:29 PM INFO hudson.slaves.NodeProvisioner$2 run
Kubernetes Pod Template provisioning successfully completed. We have now 5 computer(s)
Apr 18, 2017 2:38:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminating Kubernetes instance for slave kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6
Apr 18, 2017 2:38:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminated Kubernetes instance for slave kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6
Apr 18, 2017 2:38:35 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Disconnected computer kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6
Apr 18, 2017 2:38:35 PM WARNING jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
Computer.threadPoolForRemoting [#627] for kubernetes-d1e751c682cf44258527ee001a5a84cc-1e98f0cf330c6 terminated
java.io.EOFException
    at org.jenkinsci.remoting.nio.NioChannelHub$3.run(NioChannelHub.java:617)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Apr 18, 2017 2:38:35 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
fabric8-ui/fabric8-runtime-console/PR-311 #2 completed: FAILURE
Apr 18, 2017 2:38:35 PM WARNING hudson.model.listeners.RunListener report
RunListener failed
java.util.ConcurrentModificationException
    at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
    at java.util.TreeMap$KeyIterator.next(TreeMap.java:1265)
    at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.createBuild(BuildListener.java:71)
    at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.onCompleted(BuildListener.java:49)
    at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:211)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:618)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:125)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:901)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1092)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:400)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Apr 18, 2017 2:41:27 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminating Kubernetes instance for slave kubernetes-e9706a5ff8074c468e1a35dc3516c250-1e9b6a1af6191
Apr 18, 2017 2:41:27 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminated Kubernetes instance for slave kubernetes-e9706a5ff8074c468e1a35dc3516c250-1e9b6a1af6191
Apr 18, 2017 2:41:27 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Disconnected computer kubernetes-e9706a5ff8074c468e1a35dc3516c250-1e9b6a1af6191
Apr 18, 2017 2:41:27 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
fabric8-ui/fabric8-runtime-console/master #146 completed: FAILURE
Apr 18, 2017 2:41:27 PM WARNING hudson.model.listeners.RunListener report
RunListener failed
java.util.ConcurrentModificationException
    at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
    at java.util.TreeMap$KeyIterator.next(TreeMap.java:1265)
    at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.createBuild(BuildListener.java:71)
    at io.fabric8.kubernetes.pipeline.devops.elasticsearch.BuildListener.onCompleted(BuildListener.java:49)
    at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:211)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:618)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1400(WorkflowRun.java:125)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:901)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1092)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:400)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
rawlingsj commented 7 years ago

@iocanel I don't suppose this means anything to you does it? We've started running lots of PR builds and we can have 5 running concurrently on a three node GKE cluster, which has plenty of space resources. Roughly every 12 hours we start seeing these errors and I can only bounce the jenkins master to recover.

Any clues how to investigate what's going on?

rawlingsj commented 7 years ago

Interesting comment from

We have been having this issue, and did an investigation. It seems to be failing on pipe write in the first "cd" command to get the workspace. From what we understand the underlying kubernetes-client library does not expose a stable connection. From what we understand the latch was implemented wrongly (?? We think). The latch is initialized once in the container "start" method, but a new connection is opened for each sh step. This causes the "waitQuietly" to pass through straight away without waiting, on all sh connections but the first. Once we fixed this the "OnOpen" callback in the websocket is never called, indicating the connection is not made, and the latch was only passed on thread interrupt.

https://issues.jenkins-ci.org/browse/JENKINS-40825?focusedCommentId=290693&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-290693

vikasjio commented 7 years ago

I faced this issue on AWS also. got working after increasing resources and bouncing the jenkins pod.

tettaji commented 7 years ago

@vikasjio could you elaborate? How are your resources set now to resolve the issue?

vikasjio commented 7 years ago

Hi, For me, things were working initially; but started failing once I increased loads/applications in my cluster. So, I just added 2 more VMs in my cluster, which already had 4 worker VMs of the same size.

From: tettaji [mailto:notifications@github.com] Sent: 25 July 2017 16:17 To: fabric8io/fabric8-pipeline-library Cc: Vikas Chander; Mention Subject: Re: [fabric8io/fabric8-pipeline-library] error starting build pod java.io.IOException: Pipe not connected (#206)

@vikasjiohttps://github.com/vikasjio could you elaborate? How are your resources set now to resolve the issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/fabric8io/fabric8-pipeline-library/issues/206#issuecomment-317700702, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AURkBK8Ht_1GM4eVykNXtxmKLBQwHBHwks5sRceYgaJpZM4NAcbE. "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system.

Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."

rawlingsj commented 7 years ago

This is fixed in the kubernetes-plugin https://github.com/jenkinsci/kubernetes-plugin/pull/182, just need to release a new Jenkins image with the fix included and update the pipeline library to work with 0.13-SNAPSHOT as there seems to be some breaking change with the way env vars are added to pods. I will hopefully have this done in the next day or two.

marvinosswald commented 6 years ago

@rawlingsj is this fixed in the newest release ?

just experienced it on my setup: Jenkins: Core 2.32.3 Fabric8-pipeline-library: 2.2.311 Plugins:

  Kubernetes 1.3.2 0.10-SNAPSHOT (private-70e60d4e-jamesrawlings)
  Kubernetes :: Pipeline :: DevOps Steps 1.5 1.4-SNAPSHOT (private-e8baa5c6-jamesrawlings)
  Kubernetes :: Pipeline :: Kubernetes Steps 1.5 1.4-SNAPSHOT (private-e8baa5c6-jamesrawlings)
rawlingsj commented 6 years ago

@marvinosswald I'm not involved in the fabric8 project anymore I'm afraid so not sure where things are. From memory however I think the latest kubernetes-plugin did include a number if fixes to avoid this so I think it should be fixed.