stephenashank
commented
5 years ago Hi @yurok15, I will need some more information to help debug this.
- Was the instance
ci-centos-7-pjfhx1created before or after the restart? - If after, how you are provisioning agents (i.e. provisioning manually or being provisioned due to jobs queueing with the corresponding label)?
- Related: How soon after restarting are these agents being provisioned?
- What value is configured for "Node Retention Time"?
- What value is configured for the "Usage"?
- Is the option "Preemptible" checked?
Hello all. After restarting jenkins we begin experience issues with deleting agents. Jenkins kill agent after it successfully started. Here is a log messages:
INFO: Started provisioning ci-centos-7-pjfhx1 from gce-gce with 1 executors. Remaining excess workload: 0 Sep 24, 2019 9:35:09 AM com.google.jenkins.plugins.computeengine.ComputeEngineCloud lambda$getPlannedNodeFuture$0 INFO: Waiting 300000ms for node ci-centos-7-pjfhx1 to connect Sep 24, 2019 9:35:47 AM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log INFO: Launching instance: ci-centos-7-pjfhx1 Sep 24, 2019 9:35:47 AM com.google.jenkins.plugins.computeengine.ComputeEngineCloud log INFO: bootstrap
INFO: 64621ms elapsed waiting for node ci-centos-7-pjfhx1 to connect Sep 24, 2019 9:37:15 AM hudson.model.Run execute
Sep 24, 2019 9:38:09 AM com.google.jenkins.plugins.computeengine.CleanLostNodesWork terminateInstance INFO: Remote instance ci-centos-7-pjfhx1 not found locally, removing it
INFO: Ping failed. Terminating the channel ci-centos-7-pjfhx1. java.util.concurrent.TimeoutException: Ping started at 1569318073002 hasn't completed by 1569318313003 at hudson.remoting.PingThread.ping(PingThread.java:134) at hudson.remoting.PingThread.run(PingThread.java:90)
INFO: I/O error in channel ci-centos-7-pjfhx1 java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:861) at java.io.ObjectInputStream.(ObjectInputStream.java:357)
at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:49)
at hudson.remoting.Command.readFrom(Command.java:140)
at hudson.remoting.Command.readFrom(Command.java:126)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
WARNING: Failed to terminate ci-centos-7-pjfhx1 com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found { "code" : 404, "errors" : [ { "domain" : "global", "message" : "The resource 'projects/tangome-corpit/zones/europe-north1-a/instances/ci-centos-7-pjfhx1' was not found", "reason" : "notFound" } ], "message" : "The resource 'projects/tangome-corpit/zones/europe-north1-a/instances/ci-centos-7-pjfhx1' was not found" } at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146) at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113) at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321) at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1067) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352) at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469) at com.google.jenkins.plugins.computeengine.client.ComputeClient.terminateInstance(ComputeClient.java:365) at com.google.jenkins.plugins.computeengine.ComputeEngineInstance._terminate(ComputeEngineInstance.java:120) at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:67) at org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy$1$1.run(OnceRetentionStrategy.java:128) at hudson.model.Queue._withLock(Queue.java:1379) at hudson.model.Queue.withLock(Queue.java:1256) at org.jenkinsci.plugins.durabletask.executors.OnceRetentionStrategy$1.run(OnceRetentionStrategy.java:123) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)