jenkinsci / docker-plugin

Jenkins cloud plugin that uses Docker
https://plugins.jenkins.io/docker-plugin/
MIT License
487 stars 317 forks source link

Docker plugin stopped starting Docker containers #1087

Open andreasgalek opened 2 months ago

andreasgalek commented 2 months ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.440.2 OS: Linux - 5.15.0-206.153.7.1.el9uek.x86_64 Java: 11.0.23 - Red Hat, Inc. (OpenJDK 64-Bit Server VM) --- Office-365-Connector:4.21.0 Parameterized-Remote-Trigger:3.2.0 ace-editor:1.1 allure-jenkins-plugin:2.31.1 analysis-model-api:12.1.0 ansicolor:1.0.4 ant:497.v94e7d9fffa_b_9 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 apache-httpcomponents-client-5-api:5.3.1-1.0 appaloosa-plugin:1.4.12 asm-api:9.7-33.v4d23ef79fcc8 atlassian-bitbucket-server-integration:4.0.0 authentication-tokens:1.53.v1c90fd9191a_b_ authorize-project:1.7.1 azure-credentials:312.v0f3973cd1e59 azure-sdk:157.v855da_0b_eb_dc2 badge:1.9.1 basic-branch-build-strategies:81.v05e333931c7d bitbucket:241.v6d24a_57f9359 blackduck-detect:9.0.0 blueocean:1.27.11 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.11 blueocean-commons:1.27.11 blueocean-config:1.27.11 blueocean-core-js:1.27.11 blueocean-dashboard:1.27.11 blueocean-display-url:2.4.2 blueocean-events:1.27.11 blueocean-git-pipeline:1.27.11 blueocean-github-pipeline:1.27.11 blueocean-i18n:1.27.11 blueocean-jira:1.27.11 blueocean-jwt:1.27.11 blueocean-personalization:1.27.11 blueocean-pipeline-api-impl:1.27.11 blueocean-pipeline-editor:1.27.11 blueocean-pipeline-scm-api:1.27.11 blueocean-rest:1.27.11 blueocean-rest-impl:1.27.11 blueocean-web:1.27.11 bootstrap4-api:4.6.0-6 bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1152.v6f101e97dd77 build-blocker-plugin:1.7.9 build-failure-analyzer:2.5.0 build-name-setter:2.4.2 build-timeout:1.32 build-timestamp:1.0.3 build-user-vars-plugin:1.9 buildtriggerbadge:251.vdf6ef853f3f5 built-on-column:1.4 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.2.0 cloud-stats:336.v788e4055508b_ cloudbees-bitbucket-branch-source:883.v041fa_695e9c2 cloudbees-folder:6.901.vb_4c7a_da_75da_3 clover:4.14.2.596.vb_4d6475e990b_ cobertura:1.17 code-coverage-api:4.99.0 command-launcher:107.v773860566e2e commons-httpclient3-api:3.1-3 commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 compress-buildlog:1.2 conditional-buildstep:1.4.3 config-file-provider:968.ve1ca_eb_913f8c copyartifact:722.v0662a_9b_e22a_c coverage:1.13.0 credentials:1337.v60b_d7b_c7b_c9f credentials-binding:677.vdc9d38cb_254d cucumber-reports:5.8.1 custom-checkbox-parameter:1.4 dark-theme:439.vdef09f81f85e dashboard-view:2.508.va_74654f026d1 data-tables-api:2.0.3-1 description-setter:239.vd0a_6b_785f92d display-url-api:2.200.vb_9327d658781 docker-commons:439.va_3cb_0a_6a_fb_29 docker-java-api:3.3.4-86.v39b_a_5ede342c docker-plugin:1.6 docker-workflow:572.v950f58993843 dtkit-api:3.0.2 durable-task:555.v6802fe0f0b_82 echarts-api:5.5.0-1 email-ext:2.105 embeddable-build-status:467.v4a_954796e45d envinject:2.908.v66a_774b_31d93 envinject-api:1.199.v3ce31253ed13 extensible-choice-parameter:1.8.1 external-monitor-job:215.v2e88e894db_f8 extra-columns:1.26 favorite:2.208.v91d65b_7792a_c file-operations:214.v2e7dc7f25757 file-parameters:316.va_83a_1221db_a_7 folder-properties:1.2.1 font-awesome-api:6.5.1-3 forensics-api:2.4.0 generic-webhook-trigger:2.1.1 git:5.2.1 git-changelog:3.38 git-client:4.7.0 git-forensics:2.1.0 git-parameter:0.9.19 git-server:114.v068a_c7cc2574 github:1.38.0 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1785.v99802b_69816c golang:1.4 google-oauth-plugin:1.330.vf5e86021cb_ec gradle:2.10 groovy:457.v99900cb_85593 groovy-postbuild:228.vcdb_cf7265066 gson-api:2.10.1-15.v0d99f670e0a_7 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 hidden-parameter:237.v4b_df26c7a_f0e htmlpublisher:1.33 http_request:1.18 instance-identity:185.v303dc7c645f9 ionicons-api:70.v2959a_b_74e3cf jackson2-api:2.17.0-379.v02de8ec9f64c jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javadoc:243.vb_b_503b_b_45537 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.11 jenkins-multijob-plugin:627.v7c23cef20a_6a jersey2-api:2.42-147.va_28a_44603b_d5 jira:3.13 jira-steps:2.0.165.v8846cf59f3db jjwt-api:0.11.5-112.ve82dfb_224b_a_d jnr-posix-api:3.1.19-2 job-dsl:1.87 job-import-plugin:3.6 jobConfigHistory:1229.v3039470161a_d jobcopy-builder:1.5.0 joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ jquery:1.12.4-1 jquery3-api:3.7.1-2 jsch:0.2.16-86.v42e010d9484b_ json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b_a_655 junit:1265.v65b_14fa_f12f0 ldap:719.vcb_d039b_77d0d leastload:3.0.0 lockable-resources:1246.v28b_e4cc6fa_16 log-parser:2.3.3 mail-watcher-plugin:1.19 mailer:472.vf7c289a_4b_420 mapdb-api:1.0.9-40.v58107308b_7a_7 mask-passwords:173.v6a_077a_291eb_5 matrix-auth:3.2.2 matrix-project:822.824.v14451b_c0fd42 maven-artifact-choicelistprovider:1.17 maven-plugin:3.23 mercurial:1260.vdfb_723cdcc81 metrics:4.2.21-449.v6960d7c54c69 mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd momentjs:1.1.1 msbuild:1.31 mstest:1.0.5 multibranch-build-strategy-extension:48.v3dc306525d0c multibranch-scan-webhook-trigger:1.0.11 naginator:1.436.vb_e769dcb_cdf6 nant:248.vcc8a_3eec8db_a next-build-number:1.8 nexus-artifact-uploader:2.14 nexus-jenkins-plugin:3.16.510.v4d23e22cf563 node-iterator-api:55.v3b_77d4032326 nodelabelparameter:1.12.0 nunit:485.ve8a_85357320d oauth-credentials:0.646.v02b_66dc03d2e okhttp-api:4.11.0-172.vda_da_1feeb_c6e pam-auth:1.10 parameter-separator:166.vd0120849b_386 parameterized-scheduler:262.v00f3d90585cc parameterized-trigger:787.v665fcf2a_830b_ performance:957.v658a_7065b_92a_ pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-groovy-lib:704.vc58b_8890a_384 pipeline-input-step:491.vb_07d21da_1a_fb_ pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2184.v0b_358b_953e69 pipeline-model-definition:2.2184.v0b_358b_953e69 pipeline-model-extensions:2.2184.v0b_358b_953e69 pipeline-rest-api:2.34 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2184.v0b_358b_953e69 pipeline-stage-view:2.34 pipeline-utility-steps:2.16.2 plain-credentials:179.vc5cb_98f6db_38 plugin-usage-plugin:4.4 plugin-util-api:4.1.0 popper-api:1.16.1-3 popper2-api:2.11.6-4 powershell:2.1 prism-api:1.29.0-13 prometheus:2.5.3 pubsub-light:1.18 rebuild:332.va_1ee476d8f6d remote-file:1.24 resource-disposer:0.23 run-condition:1.7 saml:4.464.vea_cb_75d7f5e0 scm-api:689.v237b_6d3a_ef7f script-security:1326.vdb_c154de8669 signtool-jenkins-plugin:0.3 snakeyaml-api:2.2-111.vc6598e30cc65 sonar:2.17.2 sse-gateway:1.26 ssh-agent:346.vda_a_c4f2c8e50 ssh-credentials:337.v395d2403ccd4 ssh-slaves:2.948.vb_8050d697fec sshd:3.322.v159e91f6a_550 structs:337.v1b_04ea_4df7c8 subversion:1256.vee91953217b_6 theme-manager:215.vc1ff18d67920 thinBackup:1.19 throttle-concurrents:2.14 timestamper:1.26 token-macro:400.v35420b_922dcb_ trilead-api:2.142.v748523a_76693 uno-choice:2.8.3 urltrigger:1.02 validating-string-parameter:183.v3748e79b_9737 variant:60.v7290fc0eb_b_cd versioncolumn:243.vda_c20eea_a_8a_f view-job-filters:369.ve0513a_a_f5524 vsphere-cloud:2.27 warnings-ng:11.2.2 windows-slaves:1.8.1 workflow-aggregator:596.v8c21c963d92d workflow-api:1291.v51fd2a_625da_7 workflow-basic-steps:1049.v257a_e6b_30fb_d workflow-cps:3894.vd0f0248b_a_fc4 workflow-cps-global-lib:612.v55f2f80781ef workflow-durable-task-step:1336.v768003e07199 workflow-job:1400.v7fd111b_ec82f workflow-multibranch:773.vc4fe1378f1d5 workflow-remote-loader:1.6 workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:657.v03b_e8115821b_ workflow-support:896.v175a_a_9c5b_78f ws-cleanup:0.45 xtrigger-api:1.0 xunit:3.1.4 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Jenkins master: Oracle Linux 9 Jenkins agents: Oracle Linux 9 + Windows Server 2019

Reproduction steps

The issue cannot be reproduced. In the last couple of months, we had it twice. Symptoms:

  1. Jobs using labels defined in Docker agent templates do NOT start!
  2. No corresponding entries in the log of com.nirima.jenkins.plugins.docker!

Expected Results

The Docker plugin should react and

  1. Pick a Docker cloud in which a Docker image template with the given label is defined.
  2. Create Docker containers in the selected Docker host.
  3. Start the job in the container.

Actual Results

No reaction from the plugin. Triggered jobs wait endlessly for their execution as their labels cannot be resolved.

Anything else?

Current workaround: Restarting the complete Jenkins application solves the problem.

Are you interested in contributing a fix?

No response