Can't make plugin work over unix socket

rkosegi commented 12 months ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.433 OS: Linux - 4.15.0-135-generic Java: 11.0.21 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- analysis-model-api:11.13.0 ansicolor:1.0.4 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 apache-httpcomponents-client-5-api:5.2.2-1.0 authentication-tokens:1.53.v1c90fd9191a_b_ badge:1.9.1 blueocean:1.27.9 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.9 blueocean-commons:1.27.9 blueocean-config:1.27.9 blueocean-core-js:1.27.9 blueocean-dashboard:1.27.9 blueocean-display-url:2.4.2 blueocean-events:1.27.9 blueocean-git-pipeline:1.27.9 blueocean-github-pipeline:1.27.9 blueocean-i18n:1.27.9 blueocean-jwt:1.27.9 blueocean-personalization:1.27.9 blueocean-pipeline-api-impl:1.27.9 blueocean-pipeline-editor:1.27.9 blueocean-pipeline-scm-api:1.27.9 blueocean-rest:1.27.9 blueocean-rest-impl:1.27.9 blueocean-web:1.27.9 bootstrap5-api:5.3.2-2 bouncycastle-api:2.29 branch-api:2.1135.v8de8e7899051 build-name-setter:2.4.0 build-timeout:1.31 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.0.2 cloud-stats:320.v96b_65297a_4b_b_ cloudbees-bitbucket-branch-source:856.v04c46c86f911 cloudbees-disk-usage-simple:187.v6378d330d1d4 cloudbees-folder:6.858.v898218f3609d command-launcher:107.v773860566e2e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-94.v3e1f4a_926e49 conditional-buildstep:1.4.3 config-file-provider:959.vcff671a_4518b_ configuration-as-code:1746.vf1673cfe690a convert-to-pipeline:1.0 copyartifact:722.v0662a_9b_e22a_c credentials:1309.v8835d63eb_d8a_ credentials-binding:642.v737c34dea_6c2 dashboard-view:2.495.v07e81500c3f2 data-tables-api:1.13.8-1 delivery-pipeline-plugin:1.4.2 description-setter:1.10 disk-usage:1.2 display-url-api:2.200.vb_9327d658781 docker-build-step:2.10 docker-commons:439.va_3cb_0a_6a_fb_29 docker-java-api:3.3.1-79.v20b_53427e041 docker-plugin:1.3.0 docker-slaves:1.0.7 docker-workflow:572.v950f58993843 downstream-build-cache:1.7 durable-task:523.va_a_22cf15d5e0 echarts-api:5.4.3-1 extended-choice-parameter:376.v2e02857547b_a_ external-monitor-job:215.v2e88e894db_f8 extra-tool-installers:113.v5d95d475e985 favorite:2.4.3 font-awesome-api:6.4.2-1 forensics-api:2.3.0 git:5.2.1 git-client:4.5.0 git-server:99.va_0826a_b_cdfa_d github:1.37.3.1 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1751.v90e17c48a_6a_c gitlab-plugin:1.7.16 global-variable-string-parameter:1.2 google-login:1.8 google-metadata-plugin:0.5 google-oauth-plugin:1.318.vb_39c5db_e3041 google-play-android-publisher:4.2 google-source-plugin:0.4 gradle:2.9 h2-api:11.1.4.199-12.v9f4244395f7a_ handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 hetzner-cloud:84.v8acf5510fd35 htmlpublisher:1.32 instance-identity:185.v303dc7c645f9 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.15.3-372.v309620682326 jacoco:3.3.5 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:243.vb_b_503b_b_45537 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jenkins-design-language:1.27.9 jersey2-api:2.41-133.va_03323b_a_1396 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.87 jobConfigHistory:1229.v3039470161a_d join:1.21 jquery:1.12.4-1 jquery3-api:3.7.1-1 jsch:0.2.8-65.v052c39de79b_2 junit:1240.vf9529b_881428 kubernetes-cli:1.12.1 kubernetes-client-api:6.8.1-224.vd388fca_4db_3b_ kubernetes-credentials:0.11 mailer:463.vedf8358e006b_ mapdb-api:1.0.9-28.vf251ce40855d matrix-auth:3.2.1 matrix-project:818.v7eb_e657db_924 maven-plugin:3.23 metrics:4.2.18-442.v02e107157925 mina-sshd-api-common:2.11.0-86.v836f585d47fa_ mina-sshd-api-core:2.11.0-86.v836f585d47fa_ naginator:1.421.v85c11e2db_507 next-executions:239.v27031b_e26418 oauth-credentials:0.646.v02b_66dc03d2e okhttp-api:4.11.0-157.v6852a_a_fa_ec11 parameterized-trigger:787.v665fcf2a_830b_ performance:951.v5600a_c6422ed perfpublisher:8.09 pipeline-build-step:516.v8ee60a_81c5b_9 pipeline-config-history:1.6 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-maven:1362.vee39a_d4b_02b_1 pipeline-maven-api:1362.vee39a_d4b_02b_1 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2151.ve32c9d209a_3f pipeline-model-definition:2.2151.ve32c9d209a_3f pipeline-model-extensions:2.2151.ve32c9d209a_3f pipeline-npm:155.ve2e393062d86 pipeline-rest-api:2.34 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2151.ve32c9d209a_3f pipeline-stage-view:2.34 pipeline-timeline:1.0.3 pipeline-utility-steps:2.16.0 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.6.0 prism-api:1.29.0-9 pubsub-light:1.18 rebuild:330.v645b_7df10e2a_ resource-disposer:0.23 rich-text-publisher-plugin:1.5 robot:3.4.0 role-strategy:689.v731678c3e0eb_ run-condition:1.7 scm-api:683.vb_16722fb_b_80b_ script-security:1281.v22fb_899df1a_e snakeyaml-api:2.2-111.vc6598e30cc65 sonar:2.16.1 sse-gateway:1.26 ssh:2.6.1 ssh-agent:346.vda_a_c4f2c8e50 ssh-credentials:308.ve4497b_ccd8f4 ssh-slaves:2.916.vd17b_43357ce4 sshd:3.312.v1c601b_c83b_0e structs:325.vcb_307d2a_2782 timestamper:1.26 token-macro:384.vf35b_f26814ec toolenv:1.2 trilead-api:2.84.v72119de229b_7 variant:60.v7290fc0eb_b_cd view-job-filters:369.ve0513a_a_f5524 warnings-ng:10.5.1 workflow-aggregator:596.v8c21c963d92d workflow-api:1283.v99c10937efcb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3806.va_3a_6988277b_2 workflow-durable-task-step:1289.v4d3e7b_01546b_ workflow-job:1360.vc6700e3136f5 workflow-multibranch:756.v891d88f2cd46 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:865.v43e78cc44e0d ws-cleanup:0.45 yet-another-build-visualizer:1.16 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

OS: Ubuntu LTS Docker: 20.10.7

Reproduction steps

run jenkins controller using docker-compose:

1)

---
version: '3.2'
services:
  jenkins:
    container_name: jenkins
    image: jenkins/jenkins:lts
    restart: unless-stopped
    logging:
      driver: json-file
      options:
        tag: "{{.Name}}"
    ports:
    - 8000:8000
    - 8084:8080
    - 50000:50000
    volumes:
    - /usr/bin/docker:/usr/bin/docker:ro
    - /var/run/docker.sock:/var/run/docker.sock:rw
    - /opt/jenkins/data:/var/jenkins_home

2) Configure docker-local cloud using script from this repo

Expected Results

Docker plugin to connect to docker daemon

Actual Results

Error message that plugin can't connect: Connect to unix://localhost:2375 [localhost/127.0.0.1] failed: Connection refused (Connection refused)

Anything else?

Here are detailed logs

Nov 27, 2023 10:10:26 AM FINE com.nirima.jenkins.plugins.docker.DockerContainerWatchdog execute
Checking Docker Cloud docker-local at unix:///var/run/docker.sock
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalHttpClient doExecute
ex-0000000001 preparing request execution
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.ProtocolExec execute
ex-0000000001 target auth state: UNCHALLENGED
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.ProtocolExec execute
ex-0000000001 proxy auth state: UNCHALLENGED
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.ConnectExec execute
ex-0000000001 acquiring connection with route {}->unix://localhost:2375
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalExecRuntime acquireEndpoint
ex-0000000001 acquiring endpoint (3 MINUTES)
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager lease
ex-0000000001 endpoint lease request (3 MINUTES) [route: {}->unix://localhost:2375][total available: 0; route allocated: 0 of 2147483647; total allocated: 0 of 2147483647]
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$3 get
ex-0000000001 endpoint leased [route: {}->unix://localhost:2375][total available: 0; route allocated: 1 of 2147483647; total allocated: 1 of 2147483647]
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$3 get
ex-0000000001 acquired ep-0000000001
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalExecRuntime acquireEndpoint
ex-0000000001 acquired endpoint ep-0000000001
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.ConnectExec execute
ex-0000000001 opening connection {}->unix://localhost:2375
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalExecRuntime connectEndpoint
ep-0000000001 connecting endpoint (3000000000 NANOSECONDS)
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager connect
ep-0000000001 connecting endpoint to unix://localhost:2375 (3000000000 NANOSECONDS)
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator connect
localhost resolving remote address
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator connect
localhost resolved to [localhost/127.0.0.1]
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator connect
localhost:2375 connecting null->localhost/127.0.0.1:2375 (3000000000 NANOSECONDS)
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator connect
localhost:2375 connection to localhost/127.0.0.1:2375 failed (class java.net.ConnectException); terminating operation
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.DefaultManagedHttpClientConnection close
http-outgoing-0 close connection IMMEDIATE
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalExecRuntime discardEndpoint
ep-0000000001 endpoint closed
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.classic.InternalExecRuntime discardEndpoint
ep-0000000001 discarding endpoint
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager release
ep-0000000001 releasing endpoint
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager release
ep-0000000001 connection is not kept alive
Nov 27, 2023 10:10:26 AM FINE org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager release
ep-0000000001 connection released [route: {}->unix://localhost:2375][total available: 0; route allocated: 0 of 2147483647; total allocated: 0 of 2147483647]
Nov 27, 2023 10:10:26 AM WARNING com.nirima.jenkins.plugins.docker.DockerContainerWatchdog retrieveContainers
Unable to retrieve list of containers available on DockerCloud [name=docker-local, dockerURI=unix:///var/run/docker.sock] while reading list of containers (showAll=true, labelFilters={com.nirima.jenkins.plugins.docker.JenkinsId=8de6877e67724162b1587711f7c0950b})
org.apache.hc.client5.http.HttpHostConnectException: Connect to unix://localhost:2375 [localhost/127.0.0.1] failed: Connection refused (Connection refused)
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.base/java.net.Socket.connect(Socket.java:609)
    at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.lambda$connectSocket$0(PlainConnectionSocketFactory.java:91)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:90)
    at org.apache.hc.client5.http.socket.ConnectionSocketFactory.connectSocket(ConnectionSocketFactory.java:123)
    at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:184)
    at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:447)
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162)
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172)
    at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:96)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:115)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:191)
Caused: java.lang.RuntimeException
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:195)
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9)
    at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
    at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:202)
    at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:74)
    at com.github.dockerjava.core.exec.ListContainersCmdExec.execute(ListContainersCmdExec.java:44)
    at com.github.dockerjava.core.exec.ListContainersCmdExec.execute(ListContainersCmdExec.java:15)
    at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
    at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
    at com.nirima.jenkins.plugins.docker.DockerContainerWatchdog.retrieveContainers(DockerContainerWatchdog.java:263)
    at com.nirima.jenkins.plugins.docker.DockerContainerWatchdog.processCloud(DockerContainerWatchdog.java:208)
    at com.nirima.jenkins.plugins.docker.DockerContainerWatchdog.execute(DockerContainerWatchdog.java:148)
    at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:102)
    at java.base/java.lang.Thread.run(Thread.java:829)

Nov 27, 2023 10:10:26 AM INFO com.nirima.jenkins.plugins.docker.DockerContainerWatchdog execute
Not checking the list of nodes, as list of containers is known to be incomplete

I tried to connection from jenkins master to socket using curl, no issue:

curl  -s --unix-socket /var/run/docker.sock http://v1/containers/json

[{"Id":"a53f9b6632cfc9d3132af8beb6adf1b8477609970411884fce4327b31a254a6e","Names":["/jenkins"],"Image":"docker.ifne.eu/ci-master","ImageID":"sha256:477287f0c313d59508d47028a3b7c5c078cc0013bb1ddd72fbfaae1a0b781288","Command":"/usr/bin/tini -- /usr/local/bin/jenkins.sh","Created":1701068662,"Ports":[{"IP":"0.0.0.0","PrivatePort":50000,"PublicPort":50000,"Type":"tcp"},...

I was also able to attach strace and trace down actual attempt for connection:

[pid 26737] socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 801
[pid 26737] setsockopt(801, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
[pid 26737] setsockopt(801, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 26737] setsockopt(801, SOL_SOCKET, SO_KEEPALIVE, [0], 4) = 0
[pid 26737] fcntl(801, F_GETFL <unfinished ...>
[pid 26737] connect(801, {sa_family=AF_INET, sin_port=htons(2375), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)

Clearly, there should be AF_UNIX, but there is AF_INET.

Are you interested in contributing a fix?

Sure, happy to help if you give me pointers.

a-st commented 12 months ago

Today we started experiencing the same issue running Jenkins 2.426.1

PaulFridrick commented 12 months ago

Same here, I had to switch to a tcp connection on 127.0.0.1:2375 to work around the issue.

I also have another connection problem with a remote tcp docker socket secured via SSL (tcp://:2376) :

Exception while provisioning for label: 'xxxx', cloud='yyyy'
java.lang.IllegalArgumentException: Invalid Proxy
    at java.base/java.net.Socket.<init>(Socket.java:122)
    at org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory.createSocket(SSLConnectionSocketFactory.java:208)
    at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158)
    at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:447)
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162)
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172)
    at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:96)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:115)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:191)
    at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9)
    at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
    at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:202)
    at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:74)
    at com.github.dockerjava.core.exec.ListContainersCmdExec.execute(ListContainersCmdExec.java:44)
    at com.github.dockerjava.core.exec.ListContainersCmdExec.execute(ListContainersCmdExec.java:15)
    at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
    at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
    at com.nirima.jenkins.plugins.docker.DockerCloud.countContainersInDocker(DockerCloud.java:638)
    at com.nirima.jenkins.plugins.docker.DockerCloud.canAddProvisionedAgent(DockerCloud.java:656)
    at com.nirima.jenkins.plugins.docker.DockerCloud.provision(DockerCloud.java:394)
    at io.jenkins.docker.FastNodeProvisionerStrategy.applyToCloud(FastNodeProvisionerStrategy.java:71)
    at io.jenkins.docker.FastNodeProvisionerStrategy.apply(FastNodeProvisionerStrategy.java:41)
    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325)
    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

I don't know if this is the same bug, but the 2 issues appeared simultaneously for me. I can create another issue if necessary.

a-st commented 12 months ago

Maybe this is related to Apache HttpComponents Client 5.x API plugin? We rolled back to version 5.2.1-1.1 and the docker plugin runs as expected.

PaulFridrick commented 12 months ago

For the second issue, a workaround is to disable SSL on the remote Docker TCP socket and disable server credentials in Jenkins configuration. But this is insecure...

@a-st I will try to rollback as it seems safer, thanks !

krisstern commented 12 months ago

We will need to look into this https://plugins.jenkins.io/apache-httpcomponents-client-5-api compatibility issue to see if we could come up with a more sustainable fix, possibly over the winter holidays.

rkosegi commented 12 months ago

Maybe this is related to Apache HttpComponents Client 5.x API plugin? We rolled back to version 5.2.1-1.1 and the docker plugin runs as expected.

Confirmed this works as workaround, thanks for sharing it

rkosegi commented 12 months ago

Same here, I had to switch to a tcp connection on 127.0.0.1:2375 to work around the issue.

I also have another connection problem with a remote tcp docker socket secured via SSL (tcp://:2376) :

Exception while provisioning for label: 'xxxx', cloud='yyyy'
java.lang.IllegalArgumentException: Invalid Proxy
  at java.base/java.net.Socket.<init>(Socket.java:122)
  at org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory.createSocket(SSLConnectionSocketFactory.java:208)
  at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158)
  at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:447)
  at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162)
  at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172)
  at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142)
  at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
  at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
  at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
  at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:96)
  at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
  at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
  at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
  at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:115)
  at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
  at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
  at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)
  at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:191)
  at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9)
  at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
  at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:202)
  at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:74)
  at com.github.dockerjava.core.exec.ListContainersCmdExec.execute(ListContainersCmdExec.java:44)
  at com.github.dockerjava.core.exec.ListContainersCmdExec.execute(ListContainersCmdExec.java:15)
  at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
  at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)
  at com.nirima.jenkins.plugins.docker.DockerCloud.countContainersInDocker(DockerCloud.java:638)
  at com.nirima.jenkins.plugins.docker.DockerCloud.canAddProvisionedAgent(DockerCloud.java:656)
  at com.nirima.jenkins.plugins.docker.DockerCloud.provision(DockerCloud.java:394)
  at io.jenkins.docker.FastNodeProvisionerStrategy.applyToCloud(FastNodeProvisionerStrategy.java:71)
  at io.jenkins.docker.FastNodeProvisionerStrategy.apply(FastNodeProvisionerStrategy.java:41)
  at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325)
  at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)
  at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:92)
  at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
  at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:829)

I don't know if this is the same bug, but the 2 issues appeared simultaneously for me. I can create another issue if necessary.

This has probably different root cause, seems to be related to this

strangelookingnerd commented 12 months ago

This has probably different root cause, seems to be related to this

Maintainer of jenkinsci/apache-httpcomponents-client-5-api-plugin here. I can second @rkosegi supposition. Sorry for any inconvenience caused, but this one needs to be fixed in the httpcomponents library. I'll follow-up with a release of the plugin once there are updates available.

krisstern commented 12 months ago

Thanks @strangelookingnerd for the follow-up!

itfsw commented 11 months ago

same too

strangelookingnerd commented 11 months ago

I just released 5.2.3-1.0 of the plugin which includes the latest http client dependency version with the fix.

schreeedi commented 11 months ago

Just updated to apache-httpcomponents-client-5-api:5.2.3-1.0 but it didn't fix it for me, reverted to 5.2.1-1.1 and it works

strangelookingnerd commented 11 months ago

Digging a little deeper into the issue it seems like docker-java (which is used by this plugin) is not compatible with the latest httpclient5 versions - see https://github.com/docker-java/docker-java/pull/2262

lifeofguenter commented 11 months ago

I had a slightly different error (https://issues.jenkins.io/browse/JENKINS-72415):

java.lang.IllegalArgumentException: Invalid Proxy at java.base/java.net.Socket.(Unknown Source) at org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory.createSocket(SSLConnectionSocketFactory.java:208) at

but fortunately latest version (apache-httpcomponents-client-5-api:5.2.3-1.0) resolved the issue

roman-vynar commented 11 months ago

Same here, Docker socket works only with apache-httpcomponents-client-5-api 5.2.1-1.1 and broken with 5.2.3-1.0. Actually, that's a case for a month+ already... :(

Mai-Lapyst commented 10 months ago

Still dosnt works even on newest version of 5.3-1.0 of apache-httpcomponents-client-5-api, but now with a slightly different error for me when using unix:///var/run/docker.sock as the docker host uri. Downgrading to 5.2.1-1.1 seems still be the only option for now.

Log output

``` org.apache.hc.client5.http.HttpHostConnectException: Connect to unix://localhost:2375 [localhost/127.0.0.1] failed: Connection refused at java.base/sun.nio.ch.Net.pollConnect(Native Method) at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:547) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.lambda$connectSocket$0(PlainConnectionSocketFactory.java:91) at java.base/java.security.AccessController.doPrivileged(AccessController.java:569) at org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:90) at org.apache.hc.client5.http.socket.ConnectionSocketFactory.connectSocket(ConnectionSocketFactory.java:123) at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:184) at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:447) at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162) at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172) at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170) at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87) at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:191) Caused: java.lang.RuntimeException at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:195) at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9) at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228) at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:202) at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:74) at com.github.dockerjava.core.exec.VersionCmdExec.execute(VersionCmdExec.java:26) at com.github.dockerjava.core.exec.VersionCmdExec.execute(VersionCmdExec.java:13) at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21) at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33) at io.jenkins.docker.client.DockerAPI$DescriptorImpl.doTestConnection(DockerAPI.java:428) at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409) at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:78) at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:289) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:836) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698) at org.kohsuke.stapler.Stapler.service(Stapler.java:248) at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:163) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160) at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:76) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at jenkins.ErrorAttributeFilter.doFilter(ErrorAttributeFilter.java:29) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:154) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94) at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:569) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) at org.eclipse.jetty.server.Server.handle(Server.java:563) at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) at java.base/java.lang.Thread.run(Thread.java:840) ```

ethann commented 10 months ago

Finally fixed in just released version 5.3.1-1.0 of apache-httpcomponents-client-5-api. 🎉

rkosegi commented 9 months ago

Tested with 5.3.1-1.0 with same, successful results. Thanks to everybody who participated in debugging and resolving the issue. Closing ticket now.

shyoon-devops commented 2 months ago

I got same issue on jenkins 2.462.2 with apache-httpcomponents-client-5-api:5.4-118.v199115451c4d

So I rolled back apache-httpcomponents-client-5-api's version to 5.3.1-110.v77252fb_d4da_5 and it was fixed!

kirk-fitz commented 2 months ago

Nice one, Is rolling back the version of the apache-httpcomponents-client-5-api's just a work around? How can this be fixed?

strangelookingnerd commented 2 months ago

Nice one, Is rolling back the version of the apache-httpcomponents-client-5-api's just a work around? How can this be fixed?

Unfortunately yes. From what I can tell the problem lies in https://github.com/docker-java/docker-java (which is wrapped by this plugin) and it must be fixed there.

Previously the error was caused by either a missing dependency during runtime in docker-java as well as an incompatible change in Apache HttpClient5. The latter was rolled back and this fixed the issue at hand. With the new major version of HttpClient5 it seems the issue is re-introduced.

kirk-fitzsimons commented 2 months ago

Ah okay, should an issue be created in that repo?

shyoon-devops commented 1 month ago

Nice one, Is rolling back the version of the apache-httpcomponents-client-5-api's just a work around? How can this be fixed?

Yep, It was fixed by simply rolling back to apache-httpcomponents-client-5-api version.

MarkEWaite commented 1 month ago

The most recent version of apache-httpcomponents-client-5-api library, api 5.4-118.v199115451c4d , breaks one or more automated tests of the docker plugin.

Ubuntu 22.04 and Debian without Docker listening on port 2375 fail tests when testing with apache-httpcomponents-client-5-api 5.4-118.v199115451c4d . The same tests pass if tested with apache-httpcomponents-client-5-api 5.3.1-110.v77252fb_d4da_5

Ubuntu 22.04 with Docker listening on port 2375 passes tests with both versions of apache-httpcomponents-client-5.

The issue report that tracks that surprise is:

1103

enys commented 1 month ago

@MarkEWaite I've just updated, and seems that in not ontly fails the test but effectively can't connect to the socket in a running instance of Jenkins.

MarkEWaite commented 1 month ago

@MarkEWaite I've just updated, and seems that in not ontly fails the test but effectively can't connect to the socket in a running instance of Jenkins.

The short term workaround is to downgrade the apache-httpcomponents-client-5-api plugin. The long term solution is for an interested contributor to replace apache-httpcomponents-client-5-api with something like the HTTP libraries that are now included with the JDK.

strangelookingnerd commented 1 month ago

The short term workaround is to downgrade the apache-httpcomponents-client-5-api plugin. The long term solution is for an interested contributor to replace apache-httpcomponents-client-5-api with something like the HTTP libraries that are now included with the JDK.

Just to clarify, the fix would need to happen in https://github.com/docker-java/docker-java. Unfortunately there has been very little traction in the recent months. So there is not too much hope this is going to happen anytime soon.

A rewrite of the docker-plugin to not use docker-java or another implementation of it surely is an option but would require some serious effort.

enys commented 1 month ago

Thanks for the answers, and I can confirm that pinning the plugin did the job. And for those ending up here:

apache-httpcomponents-client-5-api:5.3.1-110.v77252fb_d4da_5

works.

kawaegle commented 1 month ago

for me only downgrading to this version works https://updates.jenkins.io/download/plugins/apache-httpcomponents-client-5-api/5.3.1-1.0/apache-httpcomponents-client-5-api.hpi

hope it help someone like me spending countless hours trying to debug the jenkins error

lessismordaunt commented 1 month ago

Can anyone provide more information w.r.t. the root cause instead of lots of talk of pinning to old versions? The original root cause identified by @rkosegi has merge requests into Apache HTTPComponents which seem to have dried up in January but the issue remains...

MarkEWaite commented 1 month ago

Can anyone provide more information w.r.t. the root cause instead of lots of talk of pinning to old versions? The original root cause identified by @rkosegi has merge requests into Apache HTTPComponents which seem to have dried up in January but the issue remains...

No, but I replied to a similar request that I'll attempt to investigate more deeply later this week or early next week so that a bug report can be submitted to the upstream repository. If you have capacity to investigate it, I'd love to hear the results of your investigation.

MarkEWaite commented 1 month ago

A new 5.4-124.v31e2987e48f4 release of Apache httpcomponents client 5 API is available that switches the library back to httpclient 5.3.1 while retaining the version number "5.4" in the version string.

jenkinsci / docker-plugin