search

jenkins-x / jx

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
https://jenkins-x.io/
Apache License 2.0
4.57k stars 786 forks source link

Jenkins X stuck waiting for next available executor #4040

Closed kshitijcode closed 4 years ago

kshitijcode commented 5 years ago

Summary

Jenkins X stuck waiting for next available executor.

Steps to reproduce the behavior

Installed jenkins x using jx install --provider=aws Imported project using jx import --no-draft from git

Tried to execute the project

Expected behavior

Jenkins X stuck waiting for next available executor

Actual behavior

Jenkins X should create a pod and execute the job.

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.147
jenkins x platform 2.0.330
Kubernetes cluster v1.11.8
kubectl            v1.14.2
helm client        Client: v2.13.1+g618447c
git                git version 2.20.1 (Apple Git-117)
Operating System   Mac OS X 10.14.4 build 18E226

Jenkins type

Kubernetes cluster

Custom k8s cluster on AWS

Operating system / Environment

MacOS

Also how does one install things like maven,pip in case of JenkinsX since I am using the --no-draft options?

I tried restarting jenkins-x by killing the pod.

I see the following lines : INFO: Jenkins is fully up and running May 22, 2019 9:21:54 AM com.cloudbees.jenkins.support.SupportPlugin writeBundle WARNING: Could not attach ''nodes/master/pipeline-timings.txt'' to support bundle java.io.IOException: test/webAutomation/master #1 did not yet start at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:928) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$PipelineTimings$1.writeTo(CpsFlowExecution.java:1902) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:309) at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:276) at com.cloudbees.jenkins.support.SupportPlugin$PeriodicWorkImpl.lambda$doRun$0(SupportPlugin.java:764) at java.lang.Thread.run(Thread.java:748)

doc-olliday commented 5 years ago

Hi, we face a similar issue on aks with a go project. The pipeline is stuck with "Waiting for next available executor". Kubernetes tries to start a pod, but this pod is Terminating immediately. A "kubectl describe pod" shows the following output:

Events:
  Type     Reason     Age   From                               Message
  ----     ------     ----  ----                               -------
  Normal   Scheduled  4s    default-scheduler                  Successfully assigned jx/go-hpszp to aks-agentpool-88549171-0
  Normal   Pulled     3s    kubelet, aks-agentpool-88549171-0  Container image "gcr.io/jenkinsxio/builder-go:0.1.706" already present on machine
  Normal   Created    2s    kubelet, aks-agentpool-88549171-0  Created container
  Warning  Failed     2s    kubelet, aks-agentpool-88549171-0  Error: failed to start container "go": Error response from daemon: No such container: b6ed2be45394a8656ba2cc481f22e98ae0f9c8d3c69808728cd463b2fe35cb66
  Normal   Pulled     2s    kubelet, aks-agentpool-88549171-0  Container image "jenkinsci/jnlp-slave:3.26-1-alpine" already present on machine
  Warning  Failed     2s    kubelet, aks-agentpool-88549171-0  Error: Error response from daemon: No such container: b6ed2be45394a8656ba2cc481f22e98ae0f9c8d3c69808728cd463b2fe35cb66
doc-olliday commented 5 years ago

Here is some more information. It's a freshly installed jx installation with Standalone Jenkins on AKS. The installation seems to be without errors. Each pipeline (project as well as environment) is stuck with "Waiting for next available executor". In an endless loop executor pods are created and terminated immediately. Having a closer look at kubernetes events, we can see following:

60m         Normal    Pulled                   pod/maven-0b54d                               Container image "gcr.io/jenkinsxio/builder-maven:0.1.719" already present on machine
60m         Warning   Failed                   pod/maven-0b54d                               Error: cannot find volume "volume-2" to mount into container "maven"
60m         Normal    Pulled                   pod/maven-0b54d                               Container image "jenkinsci/jnlp-slave:3.26-1-alpine" already present on machine
60m         Warning   Failed                   pod/maven-0b54d                               Error: cannot find volume "volume-2" to mount into container "jnlp"

'volume-2' is the secret mount for 'jenkins-docker-cfg'. This secret is in place and it is successfully mounted into a DevPod, which is working fine.

Any help or suggestions?

mkeith-eir commented 5 years ago

Getting the same thing as of today. Everything was working fine up 'til now.

We're getting it with our maven / java project as well as two separate Node JS / React projects.

mkeith-eir commented 5 years ago

@doc-olliday For what it's worth, after updating all of the plugins (except the one that said it would require any configured jobs / pipelines to be rebuilt), this is resolved for me.

doc-olliday commented 5 years ago

@mkeith-eir Many thanks for this tip. I've also updated all of the plugins and it worked for me, too. I just wonder a little bit, because the problem occurred on a freshly installed cluster on AKS, while on a freshly installed minikube (same version for jx version) hadn't the problem. But now it's working!

stepanselyuk commented 5 years ago

@mkeith-eir Yeah, the same thing with a fresh install on EKS. Thank you for the trick!

usultis commented 5 years ago

@mkeith-eir how did you update those plugins? I have the same issue with EKS. I tried to deploy Jenkins X 3 times already with the same result.

stepanselyuk commented 5 years ago

@usultis you need to login to jenkins jenkins.jx.yourdomain and then on the top right you will see notifications, you need to go to the plugins manager and on the bottom of the page select all plugins for update and check restart after completion mark.

usultis commented 5 years ago

@stepanselyuk thanks! that helped. I was thinking that these updates will not survive the restart but it helped.

thienlh commented 5 years ago

Happened to our production EKS cluster today too.

larkinandy commented 5 years ago

Happened with an on-prem cluster today. Updating plugins resolved the issue.

jenkins-x-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle stale

jenkins-x-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle rotten

jenkins-x-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten. Provide feedback via https://jenkins-x.io/community. /close

deanesmith commented 4 years ago

Hi @kshitijcode, We are closing this issue because static Jenkins masters have been deprecated and removed from Jenkins X in favor of focusing on Tekton pipelines.

This blog has more information as to the reasons why: https://jenkins-x.io/blog/2020/03/11/tekton/

If you’d like to discuss this in more detail, please join our Jenkins X channels in the Kubernetes Slack and tag any of our product leadership (Deane Smith, Viktor Farcic, Ethan Jones, Dave Conde, James Rawlings, James Strachan) and we’d be happy to discuss further.

If this is a surprise to you, consider joining the Jenkins X newsletter and other community channels to make sure that important notices like this always make it to you: https://jenkins-x.io/community/ https://jenkins-x.io/community/