search

jenkinsci / docker-plugin

Jenkins cloud plugin that uses Docker
https://plugins.jenkins.io/docker-plugin/
MIT License
489 stars 318 forks source link

Plugin keeps spinning up new containers until first container is up in JNLP mode #739

Open matejdro opened 5 years ago

matejdro commented 5 years ago

Jenkins 2.182 Docker plugin 1.1.6 Docker engine 18.09.7

I set up one container template with specific label (testtag in this case). Container uses JNLP as connection mechanism.

After starting ONE build that requires this node label, docker plugin will apparently keep spinning up new instances until first instance is booted up and ready to work.

From jenkins logs:

Finished DockerContainerWatchdog Asynchronous Periodic Work. 7 ms Asked to provision 1 slave(s) for: testtag Provisioning 'testtag' on 'docker'; Total containers: 0 (of 3) Will provision 'testtag', for label: 'testtag', in cloud: 'docker' Trying to run container for testtag Trying to run container for node testtag-0000yjuadiv9g from image: testimage Started container ID 3a899b68369d19cea4c7647f696c08a1d446e1bf897ea93b3a6e704d78014950 for node testtag-0000yjuadiv9g from image: testimage Asked to provision 1 slave(s) for: testtag Provisioning 'testtag' on 'docker'; Total containers: 1 (of 3) Will provision 'testtag', for label: 'testtag', in cloud: 'docker' Trying to run container for testtag Trying to run container for node testtag-0000yjyvz3geo from image: testimage Started container ID a7eeb03f8bfbb6a2edb4194c59dd9337e8f070bfe7350f92b803d11ecd978e85 for node testtag-0000yjyvz3geo from image: testimage Asked to provision 1 slave(s) for: testtag Provisioning 'testtag' on 'docker'; Total containers: 2 (of 3) Will provision 'testtag', for label: 'testtag', in cloud: 'docker' Trying to run container for testtag Trying to run container for node testtag-0000yk3l3txpw from image: testimage Started container ID 2997ce6c6a758f009c19602259a948db883431492e69d2b82939e2e8e39c8585 for node testtag-0000yk3l3txpw from image: testimage

Note that Asked to provision 1 slave(s) keeps repeating, even though a slave is already being provisioned. This keeps repeating until first container is booted up and job starts executing.

pjdarton commented 4 years ago

AIUI, this is because the docker-plugin spawns the container and returns the results to the Jenkins core before the JNLP connection has been established ... and unfortunately the Jenkins core does not count "under construction" nodes when it's trying to figure out how many more nodes are needed to satisfy demand, so it asks for more, and gets more ... and more...

To stop this, we'd need some code added into the docker plugin's handling of JNLP nodes so that it waited until the JNLP connection was established before it considered its job "complete" (because the docker-plugin has logic to count "under construction" nodes ... but this only applies to nodes it has yet to return to the Jenkins core).

...and this would probably be best done after #757 is resolved, because that'll deal with other race conditions that'll make this tricky.

theKBro commented 3 years ago

757 seems to be solved is there any progress here??