search

jenkinsci / docker-plugin

Jenkins cloud plugin that uses Docker
https://plugins.jenkins.io/docker-plugin/
MIT License
490 stars 319 forks source link

Template instance limit of '1' reached on cloud #655

Open lschnetler opened 6 years ago

lschnetler commented 6 years ago

If you get some troubles with docker-plugin, please report

I built the docker image from jenkins/ssh-slave and only added dotnet, git and nodejs

During the launch I am getting the following:

May 29, 2018 2:23:50 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision Asked to provision 1 slave(s) for: 1362b6c8-1780-4972-a8fe-f325dcd3bfc5 May 29, 2018 2:23:50 PM INFO com.nirima.jenkins.plugins.docker.DockerCloud canAddProvisionedSlave Not Provisioning 'jnb-dvps-gitlab.sybrin.co.za:4567/root/jenkins-runner'. Template instance limit of '1' reached on cloud 'jnb-dvps-r730'

This happens only when another build is already running using the same image.

Thanks in advance

pjdarton commented 6 years ago

My guess is that your template has an "Instance Capacity" set to 1. Try removing that setting, or increasing the number.

lschnetler commented 6 years ago

Hi.

Currently I have set both of my test jobs to use 20. When I initially tested it, it was on blank.

Both jobs are using the same image though. I also tried where the job shouldn't pull latest image

wangting66 commented 6 years ago

I have the same issue. container cap set to 2147483647, however only 1 docker runner spawns and the rest of the jobs queue.

is there config or settings handled somewhere else that will influence this?

erikhofer commented 6 years ago

I have the same issue when defining the template on job level. When I create the template in global configuration it works as expected.

pjdarton commented 6 years ago

@erikhofer Ah, that's very useful information. Can you provide a step-by-step explanation of how to reproduce the issue?

FYI Personally, I never define templates "on job level" and so I have no experience/knowledge in that area - where I work, we only ever use the global config but if you can explain in sufficient detail that I can easily reproduce the issue, I (or Nikolas) may be able to find the time to reproduce it locally, trace the bug to its source, and ultimately find how to fix it.

erikhofer commented 6 years ago

@pjdarton Basically what I did is, I created a Job that took some time (I used stress to simulate resource consumption but a simple "sleep" should be fine). I ticked "Define a Docker template", provided an image name and chose the "attach" type. I then created a copy of the the job (I assume this is the same as doing the same steps a second time). Then I started both jobs. One container is launched and the first job is executed, the second job is placed in the queue. The Jenkins container outputs the Template instance limit of '1' reached message. I also tried setting a limit instead of leaving it blank, same result.

The frist strange thing is that those jobs interact at all. As far as I understand, the two templates are distinct and only use the same image. The second issue is that the limit is somehow set to 1.

pjdarton commented 6 years ago

@erikhofer Could you do the same test, but this time make the image name different? They can both resolve to the same image, but what I want to know is if, perhaps, the docker plugin is mistaking the first template for the second based purely on the image name (i.e. the string you're specifying for the image). e.g. if one is myrepo/image and the other is myrepo/image:latest then they should both evaluate to be the same docker image even though they're two different text strings...

I have a nagging suspicion that the docker plugin code only uses a template's image name to tell them apart (as far as instance counts are concerned) and this test (of using two different image names while keeping everything else the same) would confirm this suspicion...

pjdarton commented 6 years ago

...also, if my suspicions are confirmed, I'd be much obliged if you could grab the config.xml file from either of your jobs and paste it here. e.g. if your Jenkins server's job URL is http://myserver/job/myJob1 then I'm after the contents of http://myserver/job/myJob1/config.xml as that'll tell me everything I'll need to know to re-create the issue locally (just take a look through it and make sure that it doesn't contain any passwords or other sensitive data before posting it publicly)

erikhofer commented 6 years ago

@pjdarton I'm afraid I don't have the time to investigate this any further at the moment. I don't have the original setup anymore because I replaced it with the global template configuration. At the moment I don't really need per job configuration to work, it just was the first thing I tried because the documentation didn't give any directions.

lschnetler commented 6 years ago

I did a quick test, @pjdarton and @erikhofer you're right. When I did the global defined template and run the worker on a specific host it works. I assume it is based on the image names.

I will try and test this weekend. For now I'm happy. To those who might be a little lost:

Create a template:

  1. Manage Jenkins -> Configure System -> Cloud -> Dokcer -> Docker Agent templates...
  2. Add all your information but take note on the Labels

Create the Job:

  1. Restrict where this project can be run -> insert your label here and it will start scaling
willis7 commented 6 years ago

Looks like a duplicate of #629

pjdarton commented 6 years ago

There's insufficient information on #629 that one cannot be 100% sure. I agree it could be but, without the other half of the problem description (the cloud config) we can't be sure.

venkatreddyjanga commented 4 years ago

I have a few questions regarding docker plugin: Version = 18.06.1-ce, API Version = 1.38 docker plugin version: 1.1.5 Jenkins version: 2.150.1 How many container we can spin-up at a time. it is based on template name or based on the image.

scenario1: x is label and x1 is imaged name set the limit for 10 containers. scenario2: Y is label name and x1 is image name set the limit for 10 conatiners.

for above case how many containers will spin-up? Is this docker container will spin up based on the cpu and memory?

I am facing issue i set limit to 10 containers but still builds are going to queue state.

I configured template globally.

Can you please help on this.