search

jenkinsci / docker-plugin

Jenkins cloud plugin that uses Docker
https://plugins.jenkins.io/docker-plugin/
MIT License
489 stars 318 forks source link

Shell Commands fail to ever start #940

Open ChtrMshipper opened 1 year ago

ChtrMshipper commented 1 year ago

Jenkins and plugins versions report

Environment ```Jenkins: 2.375.3 OS: Linux - 3.10.0-1160.49.1.el7.x86_64 Java: 11.0.17 - Red Hat, Inc. (OpenJDK 64-Bit Server VM) --- PrioritySorter:4.1.0 active-directory:2.30 ant:481.v7b_09e538fcca antisamy-markup-formatter:155.v795fb_8702324 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 artifactdeployer:1.3 authentication-tokens:1.4 bootstrap4-api:4.6.0-5 bootstrap5-api:5.2.1-3 bouncycastle-api:2.27 branch-api:2.1071.v1a_188a_562481 build-timeout:1.28 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.8.1 cloudbees-folder:6.800.v71307ca_b_986b command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-27.vb_fa_3896786a_7 credentials:1214.v1de940103927 credentials-binding:523.vd859a_4b_122e6 data-tables-api:1.12.1-4 display-url-api:2.3.7 docker-commons:1.21 docker-compose-build-step:1.0 docker-java-api:3.2.13-68.va_875df25a_b_45 docker-plugin:1.3.0 docker-slaves:1.0.7 docker-workflow:563.vd5d2e5c4007f durable-task:504.vb10d1ae5ba2f echarts-api:5.4.0-1 email-ext:2.94 external-monitor-job:203.v683c09d993b_9 font-awesome-api:6.2.1-1 git:5.0.0 git-client:4.1.0 git-server:99.va_0826a_b_cdfa_d github:1.37.0 github-api:1.303-400.v35c2d8258028 github-branch-source:1701.v00cc8184df93 gradle:2.3 instance-identity:142.v04572ca_5b_265 ionicons-api:31.v4757b_6987003 jackson2-api:2.14.2-319.v37853346a_229 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:63.v62d2fd4b_4793 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.16-2 jquery3-api:3.6.1-2 jsch:0.1.55.61.va_e9ee26616e7 junit:1177.v90374a_ef4d09 ldap:659.v8ca_b_a_fe79fa_d lockable-resources:1123.v4002ee23c671 mailer:448.v5b_97805e3767 mapdb-api:1.0.9-28.vf251ce40855d matrix-auth:3.1.6 matrix-project:785.v06b_7f47b_c631 mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a okhttp-api:4.10.0-125.v3593b_a_f8c97b_ pam-auth:1.10 pipeline-build-step:2.18.1 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:629.vb_5627b_ee2104 pipeline-input-step:466.v6d0a_5df34f81 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2118.v31fd5b_9944b_5 pipeline-model-definition:2.2118.v31fd5b_9944b_5 pipeline-model-extensions:2.2118.v31fd5b_9944b_5 pipeline-rest-api:2.31 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5 pipeline-stage-view:2.31 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:2.20.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 resource-disposer:0.21 scm-api:631.v9143df5b_e4a_a script-security:1229.v4880b_b_e905a_6 snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 ssh-credentials:305.v8f4381501156 ssh-slaves:2.854.v7fd446b_337c9 sshd:3.275.v9e17c10f2571 structs:324.va_f5d6774f3a_d subversion:2.17.0 throttle-concurrents:2.11 timestamper:1.22 token-macro:321.vd7cc1f2a_52c8 trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1208.v0cc7c6e0da_9e workflow-basic-steps:1010.vf7a_b_98e847c1 workflow-cps:3618.v13db_a_21f0fcf workflow-durable-task-step:1234.v019404b_3832a workflow-job:1268.v6eb_e2ee1a_85a workflow-multibranch:733.v109046189126 workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.44 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Centos7

Reproduction steps

Jenkins pipeline jobs fail to spawn shell commands if they were previously in a suspended state when using a docker multibranch pipeline. Please note, this only happens when the job enters a "Suspended" state and keeps building any ways.

Not sure how to provide exact details here as I am still learning a lot about jenkins, but here is the problem.

Jobs that enter a suspended state fail to execute shell commands inside the container.

From what I can tell there is some sort of IO failure with the spawning of the container.

Jenkins logs here Logs here:


Cached connection io.jenkins.docker.client.DockerAPI$SharableDockerClient@57bfaad6 to DockerClientParameters{dockerUri='tcp://localhost:7504', credentialsId='null', readTimeoutInMsOrNull=100000, connectTimeoutInMsOrNull=100000}
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerCloud canAddProvisionedAgent
Provisioning 'teldev/centos7-builder:latest' number 1 (of 2) on 'docker-uat1'; Total containers: 0 (of 2)
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision
Will provision 'teldev/centos7-builder:latest', for label: 'null', in cloud: 'docker-uat1'
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Trying to run container for image "teldev/centos7-builder:latest"
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Trying to run container for node centos7-builder-jekins-slave1-002uuz9748r51 from image: teldev/centos7-builder:latest
Feb 23, 2023 12:57:53 AM INFO com.nirima.jenkins.plugins.docker.DockerTemplate doProvisionNode
Started container ID 922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f for node centos7-builder-jekins-slave1-002uuz9748r51 from image: teldev/centos7-builder:latest
Feb 23, 2023 12:57:54 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from centos7-builder-jekins-slave1-002uuz9748r51 (922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f): Feb 23, 2023 12:57:54 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Using /var/lib/jenkins/agent.log as an agent error log destination; output log will not be generated
Feb 23, 2023 12:57:54 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from centos7-builder-jekins-slave1-002uuz9748r51 (922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f): channel started
Feb 23, 2023 12:57:56 AM INFO io.jenkins.docker.client.DockerMultiplexedInputStream readInternal
stderr from centos7-builder-jekins-slave1-002uuz9748r51 (922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f): WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by jenkins.slaves.StandardOutputSwapper$ChannelSwapper to constructor java.io.FileDescriptor(int)
WARNING: Please consider reporting this to the maintainers of jenkins.slaves.StandardOutputSwapper$ChannelSwapper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Feb 23, 2023 12:58:02 AM INFO hudson.slaves.NodeProvisioner update
Image of teldev/centos7-builder:latest provisioning successfully completed. We have now 6 computer(s)
Feb 23, 2023 12:58:44 AM INFO hudson.slaves.SlaveComputer tryReconnect
Attempting to reconnect LTWC Dev Server 1
Feb 23, 2023 12:59:22 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Disconnected computer for node 'centos7-builder-jekins-slave1-002uuz9748r51'.
Feb 23, 2023 12:59:22 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread run
I/O error in channel centos7-builder-jekins-slave1-002uuz9748r51
java.net.SocketException: Socket closed
    at java.base/java.net.SocketInputStream.socketRead0(Native Method)
    at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
    at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
    at io.jenkins.docker.client.DockerMultiplexedInputStream.readInternal(DockerMultiplexedInputStream.java:51)
    at io.jenkins.docker.client.DockerMultiplexedInputStream.read(DockerMultiplexedInputStream.java:33)
    at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:94)
    at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:74)
    at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:105)
    at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)

Feb 23, 2023 12:59:22 AM INFO io.jenkins.docker.DockerTransientNode$1 println
Removed Node for node 'centos7-builder-jekins-slave1-002uuz9748r51'.

Build log here:

[2023-02-23T00:57:51.862Z] Started by user Shipper, Michael
[2023-02-23T00:57:51.887Z] [Pipeline] Start of Pipeline
[2023-02-23T00:57:51.924Z] [Pipeline] library
[2023-02-23T00:57:51.933Z] Loading library teldev_build_tools@master
[2023-02-23T00:57:51.934Z] No credentials specified
[2023-02-23T00:57:51.934Z]  > /bin/git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@libs/b874e508a693557e2c5b7ccce06797e7c000ab4bef3a2f21cb534bf208455a8e/.git # timeout=10
[2023-02-23T00:57:51.941Z] Fetching changes from the remote Git repository
[2023-02-23T00:57:51.942Z]  > /bin/git config remote.origin.url ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-it-jenkins-sanebuild.git # timeout=10
[2023-02-23T00:57:51.948Z] Fetching upstream changes from ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-it-jenkins-sanebuild.git
[2023-02-23T00:57:51.948Z]  > /bin/git --version # timeout=10
[2023-02-23T00:57:51.955Z]  > git --version # 'git version 1.8.3.1'
[2023-02-23T00:57:51.955Z]  > /bin/git fetch --tags --progress ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-it-jenkins-sanebuild.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2023-02-23T00:57:52.183Z]  > /bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10
[2023-02-23T00:57:52.189Z] Checking out Revision 09d77cf0b427b671ead030a77dab2cb6b03a7b70 (refs/remotes/origin/master)
[2023-02-23T00:57:52.189Z]  > /bin/git config core.sparsecheckout # timeout=10
[2023-02-23T00:57:52.205Z]  > /bin/git checkout -f 09d77cf0b427b671ead030a77dab2cb6b03a7b70 # timeout=10
[2023-02-23T00:57:52.212Z] Commit message: "fixed bad var name"
[2023-02-23T00:57:52.213Z]  > /bin/git rev-list --no-walk 09d77cf0b427b671ead030a77dab2cb6b03a7b70 # timeout=10
[2023-02-23T00:57:53.084Z] [Pipeline] node
[2023-02-23T00:57:57.391Z] Running on centos7-builder-jekins-slave1-002uuz9748r51 on docker-uat1 in /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools
[2023-02-23T00:57:57.394Z] [Pipeline] {
[2023-02-23T00:57:57.477Z] [Pipeline] isUnix
[2023-02-23T00:57:57.483Z] [Pipeline] withEnv
[2023-02-23T00:57:57.485Z] [Pipeline] {
[2023-02-23T00:57:57.502Z] [Pipeline] sh
[2023-02-23T00:58:01.164Z] + docker inspect -f . teldev/debian-min-build
[2023-02-23T00:58:01.164Z] .
[2023-02-23T00:58:01.681Z] [Pipeline] }
[2023-02-23T00:58:01.702Z] [Pipeline] // withEnv
[2023-02-23T00:58:01.711Z] [Pipeline] withDockerContainer
[2023-02-23T00:58:02.608Z] centos7-builder-jekins-slave1-002uuz9748r51 on docker-uat1 seems to be running inside container 922fc12965a30123e8e17d9335b02d71067a601a210c2623a6e5488b13b0456f
[2023-02-23T00:58:02.807Z] but /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools could not be found among [/var/cache/yum, /run/docker.sock, /var/cache/mock]
[2023-02-23T00:58:02.807Z] but /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@tmp could not be found among [/var/cache/yum, /run/docker.sock, /var/cache/mock]
[2023-02-23T00:58:03.287Z] $ docker run -t -d -u 798:798 --tmpfs=/var/lib/jenkins/.gnupg:rw,exec,mode=0700,size=10m,uid=798,gid=798 -w /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools -v /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools:/var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools:rw,z -v /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@tmp:/var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** teldev/debian-min-build cat
[2023-02-23T00:58:03.781Z] $ docker top 16c37bfc43f0e057515fef5cea590671fd34577211be9c3280d9fbcec7a39719 -eo pid,comm
[2023-02-23T00:58:03.983Z] [Pipeline] {
[2023-02-23T00:58:04.036Z] [Pipeline] withEnv
[2023-02-23T00:58:04.042Z] [Pipeline] {
[2023-02-23T00:58:04.071Z] [Pipeline] stage
[2023-02-23T00:58:04.076Z] [Pipeline] { (clone git branch)
[2023-02-23T00:58:04.115Z] [Pipeline] checkout
[2023-02-23T00:58:14.600Z] No credentials specified
[2023-02-23T00:58:14.600Z] Warning: JENKINS-30600: special launcher org.jenkinsci.plugins.docker.workflow.WithContainerStep$Decorator$1@20fbff43; decorates RemoteLauncher[hudson.remoting.Channel@193b6b15:centos7-builder-jekins-slave1-002uuz9748r51] will be ignored (a typical symptom is the Git executable not being run inside a designated container)
[2023-02-23T00:58:14.643Z] Cloning the remote Git repository
[2023-02-23T00:58:14.734Z] Cloning repository ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git
[2023-02-23T00:58:14.845Z]  > /bin/git init /var/lib/jenkins/workspace/debian/build-tools/charter-rpm-tools # timeout=10
[2023-02-23T00:58:14.981Z] Fetching upstream changes from ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git
[2023-02-23T00:58:14.981Z]  > /bin/git --version # timeout=10
[2023-02-23T00:58:14.986Z]  > git --version # 'git version 1.8.3.1'
[2023-02-23T00:58:14.986Z]  > /bin/git fetch --tags --progress ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2023-02-23T00:58:27.602Z]  > /bin/git config remote.origin.url ssh://git@bitbucket.corp.chartercom.com:7999/itccc/charter-rpm-tools.git # timeout=10
[2023-02-23T00:58:27.608Z]  > /bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2023-02-23T00:58:30.912Z] Avoid second fetch
[2023-02-23T00:58:30.962Z] Checking out Revision 4b93166c6d7431c35d6a2d217016b979f9b9fc10 (refs/remotes/origin/release/dev)
[2023-02-23T00:58:30.953Z]  > /bin/git rev-parse refs/remotes/origin/release/dev^{commit} # timeout=10
[2023-02-23T00:58:31.047Z]  > /bin/git config core.sparsecheckout # timeout=10
[2023-02-23T00:58:31.059Z]  > /bin/git checkout -f 4b93166c6d7431c35d6a2d217016b979f9b9fc10 # timeout=10
[2023-02-23T00:58:32.694Z] Commit message: "rolling in priority fix"
[2023-02-23T00:58:32.738Z]  > /bin/git rev-list --no-walk 4b93166c6d7431c35d6a2d217016b979f9b9fc10 # timeout=10
[2023-02-23T00:58:32.889Z] [Pipeline] }
[2023-02-23T00:58:32.912Z] [Pipeline] // stage
[2023-02-23T00:58:32.921Z] [Pipeline] stage
[2023-02-23T00:58:32.922Z] [Pipeline] { (Build)
[2023-02-23T00:58:32.960Z] [Pipeline] sh
[2023-02-23T00:59:00.301Z] Aborted by Shipper, Michael
[2023-02-23T00:59:00.344Z] Sending interrupt signal to process
[2023-02-23T00:59:15.303Z] Click here to forcibly terminate running steps
[2023-02-23T00:59:20.345Z] After 20s process did not stop
[2023-02-23T00:59:20.352Z] [Pipeline] }
[2023-02-23T00:59:20.378Z] [Pipeline] // stage
[2023-02-23T00:59:20.388Z] [Pipeline] stage
[2023-02-23T00:59:20.389Z] [Pipeline] { (install local copy of charter-rpm-tools)
[2023-02-23T00:59:20.403Z] Stage "install local copy of charter-rpm-tools" skipped due to earlier failure(s)
[2023-02-23T00:59:20.406Z] [Pipeline] }
[2023-02-23T00:59:20.437Z] [Pipeline] // stage
[2023-02-23T00:59:20.447Z] [Pipeline] stage
[2023-02-23T00:59:20.449Z] [Pipeline] { (push package)
[2023-02-23T00:59:20.463Z] Stage "push package" skipped due to earlier failure(s)
[2023-02-23T00:59:20.466Z] [Pipeline] }
[2023-02-23T00:59:20.496Z] [Pipeline] // stage
[2023-02-23T00:59:20.506Z] [Pipeline] stage
[2023-02-23T00:59:20.508Z] [Pipeline] { (Declarative: Post Actions)
[2023-02-23T00:59:20.528Z] [Pipeline] cleanWs
[2023-02-23T00:59:20.580Z] [WS-CLEANUP] Deleting project workspace...
[2023-02-23T00:59:20.580Z] [WS-CLEANUP] Deferred wipeout is used...
[2023-02-23T00:59:20.677Z] [WS-CLEANUP] done
[2023-02-23T00:59:20.682Z] [Pipeline] }
[2023-02-23T00:59:20.706Z] [Pipeline] // stage
[2023-02-23T00:59:20.711Z] [Pipeline] }
[2023-02-23T00:59:20.734Z] [Pipeline] // withEnv
[2023-02-23T00:59:20.740Z] [Pipeline] }
[2023-02-23T00:59:20.746Z] $ docker stop --time=1 16c37bfc43f0e057515fef5cea590671fd34577211be9c3280d9fbcec7a39719
[2023-02-23T00:59:22.301Z] $ docker rm -f --volumes 16c37bfc43f0e057515fef5cea590671fd34577211be9c3280d9fbcec7a39719
[2023-02-23T00:59:22.446Z] [Pipeline] // withDockerContainer
[2023-02-23T00:59:22.452Z] [Pipeline] }
[2023-02-23T00:59:22.485Z] [Pipeline] // node
[2023-02-23T00:59:22.495Z] [Pipeline] End of Pipeline
[2023-02-23T00:59:22.504Z] Finished: ABORTED

Expected Results

I expect the shell commands to run

Actual Results

No shell command is ever run.

Anything else?

If I hop into the container jenkins spawned, the docker is not running any commands.

just "cat" and "/bin/bash" Waiting for input.

ChtrMshipper commented 1 year ago

As a note, the command failed to spawn so I terminated the build.