Defunct processes are left in jenkins/agent container

Abramo-Bagnara commented 1 year ago

Jenkins and plugins versions report

Jenkins: 2.378 OS: Linux - 5.15.0-53-generic

ace-editor:1.1 analysis-model-api:10.20.0 antisamy-markup-formatter:155.v795fb_8702324 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 authentication-tokens:1.4 bootstrap4-api:4.6.0-5 bootstrap5-api:5.2.1-3 bouncycastle-api:2.26 branch-api:2.1051.v9985666b_f6cc caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.8.0 cloudbees-bitbucket-branch-source:791.vb_eea_a_476405b cloudbees-folder:6.795.v3e23d3c6f194 command-launcher:90.v669d7ccb7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b commons-text-api:1.10.0-27.vb_fa_3896786a_7 credentials:1214.v1de940103927 credentials-binding:523.vd859a_4b_122e6 data-tables-api:1.12.1-4 display-url-api:2.3.6 docker-commons:1.21 docker-java-api:3.2.13-37.vf3411c9828b9 docker-plugin:1.2.10 durable-task:501.ve5d4fc08b0be echarts-api:5.4.0-1 eclair:3.12.0.0 email-ext:2.92 font-awesome-api:6.2.0-3 forensics-api:1.16.0 git:4.13.0 git-client:3.13.0 git-server:99.va_0826a_b_cdfa_d github:1.36.0 github-api:1.303-400.v35c2d8258028 github-branch-source:1696.v3a_7603564d04 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 instance-identity:116.vf8f487400980 ionicons-api:31.v4757b_6987003 jackson2-api:2.13.4.20221013-295.v8e29ea_354141 jakarta-activation-api:2.0.1-2 jakarta-mail-api:2.0.1-2 javax-activation-api:1.2.0-5 javax-mail-api:1.6.2-8 jaxb:2.3.7-1 jdk-tool:63.v62d2fd4b_4793 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.15-2 jquery3-api:3.6.1-2 jsch:0.1.55.61.va_e9ee26616e7 junit:1160.vf1f01a_a_ea_b_7f ldap:2.12 locale:204.v2a_f305fe7e9d lockable-resources:2.18 login-theme:1.1 mailer:438.v02c7f0a_12fa_4 matrix-auth:3.1.5 matrix-project:785.v06b_7f47b_c631 metrics:4.2.10-405.v60a_9cc74e923 mina-sshd-api-common:2.9.1-44.v476733c11f82 mina-sshd-api-core:2.9.1-44.v476733c11f82 momentjs:1.1.1 okhttp-api:4.9.3-108.v0feda04578cf pam-auth:1.10 pipeline-build-step:2.18 pipeline-github-lib:38.v445716eaedda pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:613.v9c41a_160233f pipeline-input-step:456.vd8a_957db_5b_e9 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2118.v31fd5b_9944b_5 pipeline-model-definition:2.2118.v31fd5b_9944b_5 pipeline-model-extensions:2.2118.v31fd5b_9944b_5 pipeline-rest-api:2.27 pipeline-stage-step:296.v5f6908f017a_5 pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5 pipeline-stage-view:2.27 plain-credentials:139.ved2b_9cf7587b plugin-util-api:2.18.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 prism-api:1.29.0-1 resource-disposer:0.20 role-strategy:562.v44e9a_e828d0e scm-api:621.vda_a_b_055e58f7 script-security:1190.v65867a_a_47126 simple-theme-plugin:136.v23a_15f86c53d snakeyaml-api:1.33-90.v80dcb_3814d35 ssh-credentials:305.v8f4381501156 ssh-slaves:2.854.v7fd446b_337c9 sshd:3.249.v2dc2ea_416e33 structs:324.va_f5d6774f3a_d timestamper:1.21 token-macro:308.v4f2b_ed62b_b_16 trilead-api:2.72.v2a_3236754f73 variant:59.vf075fe829ccb warnings-ng:9.20.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1200.v8005c684b_a_c6 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:3536.vb_8a_6628079d5 workflow-durable-task-step:1210.va_1e5d77e122b workflow-job:1254.v3f64639b_11dd workflow-multibranch:716.vc692ae52371b workflow-scm-step:400.v6b_89a1317c9a workflow-step-api:639.v6eca_cd8c04aa workflow-support:839.v35e2736cfd5c ws-cleanup:0.43

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux

Reproduction steps

Docker Host URI: unix:///var/run/docker.sock
Connect method: Attach Docker Container
Do a build

Expected Results

Orphan build processes should be waited (like e.g. dumb-init does)

Actual Results

The docker image currently has no process that wait orphan processes and this leads to processed marked as defunct

Anything else?

No response

dduportal commented 1 year ago

Hi @Abramo-Bagnara could you give more details and/or a reproduction case?

I'm asking because since there are no entrypoint on the base jenkins/agent image, the process tree only depends on your setup so it is hard to determine what could be done to help.

You have different solutions in short term:

Try with the jenkins/inbound-agent instead (which is recommended) because it provides a default entrypoint script which uses exec to replace itsefl with the agent java command. The agent java command knows how to handle signals so it should not keep any defunct process.
Use the --init flag for docker to specify yourself tini (since you setup the whole thing)
Customize the image to add your own process tree wrapper and use it

Abramo-Bagnara commented 1 year ago

I'm a bit surprised by your answer: I had understood from https://plugins.jenkins.io/docker-plugin/ that using "Attach Docker container" connect method I don't need anything on Docker image except JDK. Perhaps I should not use this method as it is troublesome in some way and the other connect methods are preferable (also when Jenkins master and Docker daemon are on the same machine)?

dduportal commented 1 year ago

🤔 to be quite transparent, I have no idea how this plugin works because I don't use it at all (besides it needs maintainers).

But whatever method you are using for this plugin, the entrypoint is specified by the docker-plugin itself. it has nothing to do with the image here (hence my question).

Can you try the "inbound" method (formerly known as "JNLP") - https://plugins.jenkins.io/docker-plugin/#plugin-content-launch-via-jnlp and see if you still see the same "defunct" behavior?

We could look at what the plugin does with the method you mentionned (https://plugins.jenkins.io/docker-plugin/#plugin-content-launch-attached) .

Abramo-Bagnara commented 1 year ago

I have no idea how this plugin works because I don't use it at all (besides it needs maintainers).

I'm curious, what do you use (or would use) instead to run Jenkins jobs in ephemeral docker containers in a transparent way?

dduportal commented 1 year ago

I have no idea how this plugin works because I don't use it at all (besides it needs maintainers).

I'm curious, what do you use (or would use) instead to run Jenkins jobs in ephemeral docker containers in a transparent way?

Depends on the problem I would have to solve (e.g. what is the goal to have ephemeral containers?).

My "de facto" default is to use an SSH permanent agent because it's really easy and costless to manage. Example here with docker-compose: https://github.com/dduportal/jenkins-examples/tree/main/docker (I always rely on Jenkins Configuration as code).

From this example, I tend to build a custom docker image on top of the jenkins-ssh-agent to add the build tooling required by my developers.
Besides "easy maintenance", it makes it really easy to provide a reproductible environment for developers: if they cannot build because of a tooling error, they can try this image locally quite easily (docker run ... && docker exec --user=jenkins -ti <cid> bash)

dduportal commented 1 year ago

Closing the issue as it's not related to the image itself.

Feel free to reopen if you have more details pointing to this image

jenkinsci / docker-agent