search

KostyaSha / yet-another-docker-plugin

Jenkins Yet Another Docker Plugin
https://plugins.jenkins.io/yet-another-docker-plugin
MIT License
83 stars 48 forks source link

Container digest fails #161

Open flipy opened 7 years ago

flipy commented 7 years ago

Using docker:

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-16.el7.centos.x86_64
 Go version:      go1.7.4
 Git commit:      3a094bd/1.12.6
 Built:           Fri Apr 14 13:46:13 2017
 OS/Arch:         linux/amd64

And Jenkins 2.46.2 and YADP 0.1.0-rc36, running from the official docker container.

When invoking the creation of a new docker slave, it fails with the following error:

May 14 12:13:43 tpabcp5psdocker01 docker: WARNING: Unexpected exception encountered while provisioning agent evarga/jenkins-slave
May 14 12:13:43 tpabcp5psdocker01 docker: com.github.kostyasha.yad_docker_java.com.github.dockerjava.api.exception.DockerClientException: Could not pull image: Digest: sha256:3384241d2bc8846987170b54f1fb1ba68359f55936ea815ca36d8bac40154630
May 14 12:13:43 tpabcp5psdocker01 docker: at com.github.kostyasha.yad_docker_java.com.github.dockerjava.core.command.PullImageResultCallback.awaitSuccess(PullImageResultCallback.java:50)
May 14 12:13:43 tpabcp5psdocker01 docker: at com.github.kostyasha.yad.commons.DockerPullImage.exec(DockerPullImage.java:111)
May 14 12:13:43 tpabcp5psdocker01 docker: at com.github.kostyasha.yad.DockerCloud.provisionWithWait(DockerCloud.java:226)
May 14 12:13:43 tpabcp5psdocker01 docker: at com.github.kostyasha.yad.DockerCloud.lambda$provision$0(DockerCloud.java:133)
May 14 12:13:43 tpabcp5psdocker01 docker: at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
May 14 12:13:43 tpabcp5psdocker01 docker: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
May 14 12:13:43 tpabcp5psdocker01 docker: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
May 14 12:13:43 tpabcp5psdocker01 docker: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
May 14 12:13:43 tpabcp5psdocker01 docker: at java.lang.Thread.run(Thread.java:745)

The docker container can be pulled and started from command line:

[root@tpabcp5psdocker01 ~]# docker pull evarga/jenkins-slave
Using default tag: latest
Trying to pull repository docker.io/evarga/jenkins-slave ...
latest: Pulling from docker.io/evarga/jenkins-slave
Digest: sha256:3384241d2bc8846987170b54f1fb1ba68359f55936ea815ca36d8bac40154630

[root@tpabcp5psdocker01 ~]# docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                                                  NAMES
aab0e6ae9eb8        evarga/jenkins-slave      "/usr/sbin/sshd -D"      6 minutes ago       Up 6 minutes        22/tcp    
```                                                             lonely_kare
KostyaSha commented 7 years ago

please post dockercloud config from global xml config

flipy commented 7 years ago

Cloud settings from config.xml:

<clouds>
    <com.github.kostyasha.yad.DockerCloud plugin="yet-another-docker-plugin@0.1.0-rc36">
      <name>tpabcp5psdocker01</name>
      <provisionedImages/>
      <templates>
        <com.github.kostyasha.yad.DockerSlaveTemplate>
          <id>13fac45b-a6fe-4415-8c63-a15459f81839</id>
          <labelString>docker</labelString>
          <launcher class="com.github.kostyasha.yad.launcher.DockerComputerJNLPLauncher">
            <launchTimeout>120</launchTimeout>
            <user>jenkins</user>
            <jvmOpts></jvmOpts>
            <slaveOpts></slaveOpts>
            <jenkinsUrl>jenkins</jenkinsUrl>
            <noCertificateCheck>false</noCertificateCheck>
          </launcher>
          <remoteFs>/home/jenkins</remoteFs>
          <mode>EXCLUSIVE</mode>
          <retentionStrategy class="com.github.kostyasha.yad.strategy.DockerOnceRetentionStrategy">
            <idleMinutes>10</idleMinutes>
            <idleMinutes defined-in="com.github.kostyasha.yad.strategy.DockerOnceRetentionStrategy">10</idleMinutes>
          </retentionStrategy>
          <numExecutors>1</numExecutors>
          <dockerContainerLifecycle>
            <image>evarga/jenkins-slave</image>
            <pullImage>
              <pullStrategy>PULL_LATEST</pullStrategy>
              <credentialsId></credentialsId>
            </pullImage>
            <createContainer>
              <command></command>
              <hostname></hostname>
              <dnsHosts class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList">
                <c class="list"/>
                <list reference="../c"/>
              </dnsHosts>
              <volumes class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList">
                <c class="list"/>
                <list reference="../c"/>
              </volumes>
              <volumesFrom class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList">
                <c class="list"/>
<list reference="../c"/>
              </volumesFrom>
              <environment class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList">
                <c class="list"/>
                <list reference="../c"/>
              </environment>
              <bindPorts></bindPorts>
              <bindAllPorts>false</bindAllPorts>
              <memoryLimit>0</memoryLimit>
              <privileged>false</privileged>
              <tty>false</tty>
              <extraHosts class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList">
                <c class="list"/>
                <list reference="../c"/>
              </extraHosts>
              <devices class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList">
                <c class="list"/>
                <list reference="../c"/>
              </devices>
              <cpusetCpus></cpusetCpus>
              <cpusetMems></cpusetMems>
              <links class="java.util.Collections$UnmodifiableRandomAccessList" resolves-to="java.util.Collections$UnmodifiableList">
                <c class="list"/>
                <list reference="../c"/>
              </links>
            </createContainer>
            <stopContainer>
              <timeout>10</timeout>
            </stopContainer>
<removeContainer>
              <removeVolumes>true</removeVolumes>
              <force>false</force>
            </removeContainer>
          </dockerContainerLifecycle>
          <nodeProperties/>
          <maxCapacity>1</maxCapacity>
        </com.github.kostyasha.yad.DockerSlaveTemplate>
      </templates>
      <containerCap>10</containerCap>
      <connector>
        <serverUrl>tcp://tpabcp5psdocker01.us.scytl.net:2375</serverUrl>
        <apiVersion>1.24</apiVersion>
        <credentialsId></credentialsId>
        <connectorType>JERSEY</connectorType>
      </connector>
    </com.github.kostyasha.yad.DockerCloud>
  </clouds>
KostyaSha commented 7 years ago

I think it fixed already. Is it still reproducible with the latest version?

bdspecht commented 7 years ago

FWIW, I noticed the same behavior and needed to remove '--signature-verification=false' from my docker daemon service config, and then restart the docker daemon.

KostyaSha commented 7 years ago

That may be the question to docker-java. Could you compare two text outputs from docker pull XXX command with and without?

bdspecht commented 7 years ago

Seems like https://github.com/docker-java/docker-java/issues/845 is the relevant bug. Let me know if you want me to paste in this info in that bug as well.

Here's the output with the sig-verification disabled:

Trying to pull repository XXX:4567/devops/jenkins-deploy ...
ansible: Pulling from XXX:4567/devops/jenkins-deploy
Digest: sha256:68111dd0797e3ebcc4d97c732efd237f166b55c8f8548df48d1ef366dbcd10f9

And sig-verification enabled:

Trying to pull repository XXX:4567/devops/jenkins-deploy ...
sha256:68111dd0797e3ebcc4d97c732efd237f166b55c8f8548df48d1ef366dbcd10f9: Pulling from XXX:4567/devops/jenkins-deploy
Digest: sha256:68111dd0797e3ebcc4d97c732efd237f166b55c8f8548df48d1ef366dbcd10f9
Status: Image is up to date for XXX:4567/devops/jenkins-deploy:ansible
KostyaSha commented 7 years ago

So instead of docker image it shows hash. Both looks the same. Could you rm image before pulling it? Second log looks like after download.

KostyaSha commented 7 years ago

@bdspecht and what yad plugin version do you use?

bdspecht commented 7 years ago

Yad plugin version: 0.1.0-rc39 Docker version: 1.12.6-32.git88a4867 on RHEL 7.3

Docker pull with sig-verification disabled, and after removing the image:

Trying to pull repository XXX:4567/devops/jenkins-deploy ...
ansible: Pulling from XXX:4567/devops/jenkins-deploy
74f0853ba93b: Already exists
c034d2d1081b: Already exists
8967c1d93494: Already exists
4aab4c14ca30: Already exists
169f5cb0438b: Already exists
837f539612fb: Already exists
f71350e040a9: Already exists
a1f424df8793: Already exists
575b84182fee: Already exists
ac96946d4018: Already exists
91dc2c5291a4: Already exists
cd53e7dc2cfb: Already exists
0cf947f15c3b: Already exists
eee26762c468: Already exists
2dbe02ccb7c9: Pull complete
d3549ff20f25: Pull complete
Digest: sha256:68111dd0797e3ebcc4d97c732efd237f166b55c8f8548df48d1ef366dbcd10f9