jenkinsci / docker-plugin

Jenkins cloud plugin that uses Docker
https://plugins.jenkins.io/docker-plugin/
MIT License
489 stars 318 forks source link

ERROR: pending-Jenkins doesn't have label docker #574

Closed jungopro closed 6 years ago

jungopro commented 6 years ago

The basics:

More environment info: I'm running jenkins master inside docker container. I'm able to communicate with the docker host via the host URI tcp://172.16.1.1:4243 from within the jenkins container and in the docker plugin I am able to see all docker images and containers currently running on the host

The issue: When trying to configure an agent with a label docker and then run a test build I'm getting the error in the subject - pending-Jenkins doesn't have label docker and no slave container is up

Additional info:

Any help is greatly appriciated

Omer

lunderhage commented 6 years ago

Wow, I have the exact same problem. Job is waiting forever for the docker agent template label that I have specified and which is Label java-on-linux is serviced by no nodes and 1 cloud. Permissions or other restrictions provided by plugins may prevent this job from running on those nodes.

ndeloof commented 6 years ago

duplicates https://github.com/jenkinsci/docker-plugin/issues/579

sporkisfaster commented 6 years ago

Still having this issue. Getting Jenkins doesn't have label testslave in any job configuration in jenkins. Tried pipeline, freestyle, etc. Tried SSH injection, manually pasting the key, JNLP, etc. Made sure there were no leading spaces in the label name. Not sure what's going on.

Jenkins: v 2.100 Docker plugin 1.1.2 Version = 17.07.0-ce, API Version = 1.31

csakaszamok commented 6 years ago

+1

wkonsek commented 6 years ago

+1 Jenkins 2.89.3 Docker plugin 1.1.2 Docker version 18.01.0-ce

ghost commented 6 years ago

+1

Lunaden commented 6 years ago

+1

PeterKneale commented 6 years ago

+1

DamonStamper commented 6 years ago

+1 Jenkins version: 2.89.4 Docker plugin version: 1.1.3 Docker engine version: 18.02.0-ce Docker engine API version: 1.36

mircea-milencianu commented 6 years ago

+1 Jenkins ver. 2.89.4 Docker plugin: 1.1.3 Docker engine version: 17.12.0-ce Docker engine API version: 1.35

yonatanelkad commented 6 years ago

+1 same issue here Jenkins ver. 2.107.1

pjdarton commented 6 years ago

Folks,

FYI this message of "Jenkins doesn't have label docker" is rather misleading (and unhelpful). Whenever Jenkins has builds in the queue and executors that are idle but which don't have the correct labels for the build (which is a pretty normal scenario where one has more than one kind of slave), Jenkins will "complain" that the idle executors don't have the label(s) it's looking for. i.e. this "error" isn't an error, it's normal.

This message can, however, be a symptom of a failure to start containers and bring them online because, in such a situation, Jenkins has plenty of time in which to "complain" that the slaves it does have available aren't any use for the builds it needs to run. I guess that's the situation for everyone who's commented here.

To solve the underlying problem in these situations, you need to go look at the Jenkins log and see if there's a big nasty exception being thrown every time the docker plugin tries to start your container - that's the usual cause of this symptom. Another potential cause is the docker container being successfully started by the plugin but failing to come online - that usually shows up as the slaves appearing in Jenkins as "(offline)", never becoming online, and eventually being disposed of by the docker plugin (that's what the idle timeout is for). That's usually caused by problems with the "connection method", e.g. using SSH to connect to the plugin where there's no network connectivity between Jenkins and the container, or with invalid login credentials etc.

In both cases, the "Jenkins doesn't have label docker" isn't much help - you need to delve into the Jenkins logs ("Manage Jenkins" -> "System Log") and see what's being logged there.

hjoab commented 6 years ago

And then what?? What is the solution being that the case??

pjdarton commented 6 years ago

As mentioned above, you look in the logs for the error(s) that tell you what containers the docker-plugin is trying to start, and why it can't do that. Then you have to resolve those errors; they're usually caused by user-error, usually by misconfiguring the docker-plugin, or docker, or the docker registry etc. Once you've got the containers starting, you then need to ensure that they're coming online - if they don't then you've got more misconfiguration to resolve.

FYI version 1.1.4 (released recently) has a "if that didn't work, don't try again for a few minutes" back-off strategy that also shows up in the configuration pages on the UI. However that's only going to show you things that would have also shown up in the log.

biozshock commented 6 years ago

@pjdarton your comment was very helpful. I think they need to bring that error message with a stacktrace instead of one in the OP message.

For me the issue was that jenkins was not able to connect to the registry to pull local image. After i changed "Pull strategy" to "Never pull" i've got it working.

tmikulin commented 6 years ago

my case was, that I was using an instance that was very low on memory, and that was causing troubles, when I used a better instance all worked just fine...

harshblog150 commented 5 years ago

@pjdarton in my case i able to test connection ,ssh and telnet localhost:4243 (container ssh port port) to the container,but in jenkins job when i restrict to that label, it shows "Label docker-slave is serviced by no nodes and 1 cloud. Permissions or other restrictions provided by plugins may prevent this job from running on those nodes".

*below are Jenkins Log: Dropped connection io.jenkins.docker.client.DockerAPI$SharableDockerClient@5f86081f to DockerClientParameters{dockerUri=unix://xx.xx.xx.xx:4243/, credentialsId=null, readTimeoutInMsOrNull=15000, connectTimeoutInMsOrNull=5000} Jun 11, 2019 9:53:42 AM INFO io.jenkins.docker.client.DockerAPI getOrMakeClient Cached connection io.jenkins.docker.client.DockerAPI$SharableDockerClient@2aeda885 to DockerClientParameters{dockerUri=unix://xx.xx.xx.xx:4243/, credentialsId=null, readTimeoutInMsOrNull=15000, connectTimeoutInMsOrNull=5000}

something wierd, dont understand, i tried JNLP method also,doesnot workout.... if you fix solution please share or any document, its crazyyy, spent long time. thanks!

pjdarton commented 5 years ago

FYI this is not a bug; this is a symptom of misconfiguration. When Jenkins has a build queued, it displays a running commentary on why it hasn't started it yet, and this commentary is often misleading when you're expecting Jenkins to auto-provision slaves itself . When a Jenkins server is configured to auto-provision slaves itself (e.g. using the docker-plugin) then the real reason why it hasn't managed to provision a slave is displayed on the Jenkins internal logs (Manage Jenkins -> System Log) and nowhere else.

If you're seeing docker nodes show up in Jenkins but fail to come online then you most likely have a mistake in how you've configured the connection between them.

The most common mistake I've seen where I work is, when configuring the SSH connector, folks don't set the "Host Key Verification Strategy" field to "Non verifying Verification Strategy". Docker contains will usually have their own (unique) host key and therefore will not be on any list that Jenkins has been told to trust. Another common mistake is telling Jenkins to SSH in using a username/password (or username & key) that the container does not accept. In both these situations, the docker-plugin will successfully start the container, but Jenkins will fail to connect to it (because the SSH connection is misconfigured), and Jenkins will then continue to tell you that it can't run your build on "Jenkins" because it doesn't have the label you asked for ("docker" in the case of the report at the top; "docker-slave" in your case).

If you're not seeing docker nodes show up at all then you've probably misconfigured the container settings themselves, and that should be detected by the plugin and cause it to temporarily disable that template. I would suggest that you go into Manage Jenkins -> Configure System, scroll down to the "Clouds" section, expand the Docker Cloud templates, and see if the one you are expecting to be active has been automatically disabled. If there's a yellow warning telling you it's disabled, there will also be a link to expand the error, and you'll then get a better idea of why your container isn't starting.

Lusandra commented 5 years ago

Trying Jenkins out and this is what I get ‘Jenkins’ doesn’t have label ‘dotnet-21’

pjdarton commented 5 years ago

Yup, that's what it does; that's not a bug; it's not an error; see my previous comment for explanation.

mweinberg-cm commented 4 years ago

We are currently evaluating Jenkins and Mesos and ran into the same issue. What @pjdarton said was very helpful and got me on the right track. In our case, we startet with a simple echo "hello" which worked fine and it stopped working when we add proper stages and steps for a job.

Solution: Make sure that the configured agents / container have enough memory available. Increasing the memory fixed it for us.

@pjdarton thanks for the elaborative answers! :+1:

Sam-SFO commented 3 years ago

I have the same problem - job is pending for long time. Both the Cloud and docker agent showed disabled - enabled them back. I can see that the container was actually deployed on the target/docker host - so something did happen. I do have adequate memory on my docker VM/4GB - I can see lot of free memory - so that likely is not an issue. I had tested the connection in the cloud setup console - worked fine. I can Curl to validate API the target host - curl http://<ip>:4243/version. I am using 2 seprate Jenkins and Docker ubuntu machines on my local home network. I have tried all image pull policies.

The system logs show "[SSH] Authentication failed".
My creds is set to SSH with id/pwd for the container (not the target host machine) - is that the correct way? I am using a public container bibinwilson/jenkins-slave:latest with Jenkins/jenkins as id/pwd - as documented. Verification - is non verifying.

I can see the containers have been launched and running in the Docker host machine.

Issue Resolved - id was wrong - case sensitive: jenkins/jenkins worked Had to debug ssh'ing directly into the container when id/pwd did not work - double checked

AntoniusGolly commented 3 years ago

I had a slightly different issue with an EC2 Agent. For me the solution was just a bit of patience. Unfortunately, when the agent is very cold (shut down, not pending) and a job with a specific label requires that agent the agent will start but without showing logs about the EC2 spinup in that location. As I had a lot of init commands added lately the spinup just took some time.

Maybe helpful to some although slightly off-topic. Can be easily checked by going to your instances table in EC2 and see if an instance is started.

pradeesh-kumar commented 3 years ago

I faced same issue. It's due to 'CPU Period in us' and 'CPU Quota in us' mentioned as insufficient value in the container configuration in Docker Agent template settings in Jenkins.

After changing the values to 10000 resolved the issue.

Jenkin Home -> Manage Config -> Cloud -> Docker-Agent -> Docker Agent templates -> Container settings

Screenshot 2021-01-12 at 4 44 33 PM
nathan-jobrain commented 1 year ago

I fixed it by adding the label 👍 "java-docker-slave" into my main node in my case built-in node and it worked