search

jenkinsci / docker-plugin

Jenkins cloud plugin that uses Docker
https://plugins.jenkins.io/docker-plugin/
MIT License
490 stars 320 forks source link

1.0 fails to pull from Docker Hub or Amazon ECR #524

Closed ben-gineer closed 6 years ago

ben-gineer commented 6 years ago

Hi @ndeloof

This is a related issue mentioned on https://github.com/jenkinsci/docker-plugin/issues/484

I've just updated to version 1.0 but experiencing an error when trying to build a project:

Oct 16, 2017 11:18:27 PM com.nirima.jenkins.plugins.docker.DockerCloud provision
INFO: Asked to provision 1 slave(s) for: docker-tmp
Oct 16, 2017 11:18:27 PM com.nirima.jenkins.plugins.docker.DockerCloud provision
INFO: Will provision 'benginear/jenkins-slave', for label: 'docker-tmp', in cloud: 'docker'
Oct 16, 2017 11:18:27 PM com.nirima.jenkins.plugins.docker.DockerCloud addProvisionedSlave
INFO: Provisioning 'benginear/jenkins-slave' number '0' on 'docker'; Total containers: '0'
Oct 16, 2017 11:18:27 PM io.jenkins.docker.DockerSlaveProvisioner pullImage
INFO: Pulling image 'benginear/jenkins-slave:latest'. This may take awhile...
Oct 16, 2017 11:18:27 PM com.nirima.jenkins.plugins.docker.DockerCloud$1 call
SEVERE: Error in provisioning; template='DockerTemplate{configVersion=2, labelString='docker-tmp', launcher=com.nirima.jenki
ns.plugins.docker.launcher.DockerComputerSSHLauncher@41526bbc, remoteFsMapping='/opt/jenkins-data', remoteFs='/home/jenkins'
, instanceCap=4, mode=NORMAL, retentionStrategy=com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy@57ed4
423, numExecutors=1, dockerTemplateBase=DockerTemplateBase{image=benginear/jenkins-slave}, removeVolumes=false, pullStrategy
=PULL_LATEST, nodeProperties=[]}' for cloud='docker'
java.lang.NullPointerException
        at com.nirima.jenkins.plugins.docker.DockerCloud.getAuthConfig(DockerCloud.java:778)
        at com.nirima.jenkins.plugins.docker.DockerCloud.setRegistryAuthentication(DockerCloud.java:749)
        at io.jenkins.docker.DockerSlaveProvisioner.pullImage(DockerSlaveProvisioner.java:91)
        at io.jenkins.docker.DockerSlaveProvisioner.runContainer(DockerSlaveProvisioner.java:55)
        at io.jenkins.docker.SSHDockerSlaveProvisioner.provision(SSHDockerSlaveProvisioner.java:63)
        at com.nirima.jenkins.plugins.docker.DockerCloud.provisionWithWait(DockerCloud.java:397)
        at com.nirima.jenkins.plugins.docker.DockerCloud.access$000(DockerCloud.java:80)
        at com.nirima.jenkins.plugins.docker.DockerCloud$1.call(DockerCloud.java:298)
        at com.nirima.jenkins.plugins.docker.DockerCloud$1.call(DockerCloud.java:295)
        at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

I was getting a similar error when trying to use the Amazon ECR credentials. However, I'm seeing the same issue when pulling from Docker Hub hosted repositories.

I am running: Jenkins 2.85 Amazon ECR Plugin 1.6 Credentials Plugin 2.1.16 Docker API Plugin 3.0.13 Docker Commons Plugin 1.9 JClouds Plugin 2.14 Docker Version = 17.09.0-ce, API Version = 1.32

Please let me know how to debug this issue?

Thanks

ndeloof commented 6 years ago

I'm setting up an ECR instance to reproduce this issue. from stacktrace, it seems ECR credentials didn't produced an authentication token, which is weird.

ndeloof commented 6 years ago

Shouldn't your docker image ID be something like 1234.dkr.ecr.us-east-1.amazonaws.com/jenkins-slave ?

ben-gineer commented 6 years ago

Yes - docker ID for my ECR repository is:

INFO:Pulling image 'xxx.dkr.ecr.eu-west-1.amazonaws.com/jenkins-slave:latest'. This may take awhile...

(see comment on https://github.com/jenkinsci/docker-plugin/issues/484)

The benginear/jenkins-slave was my DockerHub test - which also fails with a similar error.

Thanks

ndeloof commented 6 years ago

Should be fixed by 577fa4838a6df1d4d310075b64a3755414c6017e I plan to release 1.0.1 asap

ben-gineer commented 6 years ago

Thanks for the quick turnaround. Will test and let you know.

ben-gineer commented 6 years ago

Hi

I've updated to 1.0.1, updated the Docker API Plugin to 3.0.14, and retried pulling image from both Docker Hub and Amazon ECR repositories. However, I'm still seeing the same Auth error:

INFO: Pulling image '176207018055.dkr.ecr.eu-west-1.amazonaws.com/content-delivery/jenkins-slave:latest'. This may take awhile...
Oct 17, 2017 11:16:31 PM com.nirima.jenkins.plugins.docker.DockerCloud$1 call
SEVERE: Error in provisioning; template='DockerTemplate{configVersion=2, labelString='docker-tmp', launcher=com.nirima.jenkins.plugins.docker.launcher.DockerComputerSSHLauncher@d3c8316, remoteFsMapping='/opt/jenkins-data', remoteFs='/home/jenkins', instanceCap=4, mode=NORMAL, retentionStrategy=com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy@4ecc1d39, numExecutors=1, dockerTemplateBase=DockerTemplateBase{image=xxx.dkr.ecr.eu-west-1.amazonaws.com/content-delivery/jenkins-slave}, removeVolumes=false, pullStrategy=PULL_LATEST, nodeProperties=[]}' for cloud='docker'
java.lang.NullPointerException
        at com.nirima.jenkins.plugins.docker.DockerCloud.getAuthConfig(DockerCloud.java:761)
        at com.nirima.jenkins.plugins.docker.DockerCloud.setRegistryAuthentication(DockerCloud.java:744)
        at io.jenkins.docker.DockerSlaveProvisioner.pullImage(DockerSlaveProvisioner.java:91)
        at io.jenkins.docker.DockerSlaveProvisioner.runContainer(DockerSlaveProvisioner.java:55)
        at io.jenkins.docker.SSHDockerSlaveProvisioner.provision(SSHDockerSlaveProvisioner.java:64)
        at com.nirima.jenkins.plugins.docker.DockerCloud.provisionWithWait(DockerCloud.java:392)
        at com.nirima.jenkins.plugins.docker.DockerCloud.access$000(DockerCloud.java:75)
        at com.nirima.jenkins.plugins.docker.DockerCloud$1.call(DockerCloud.java:293)
        at com.nirima.jenkins.plugins.docker.DockerCloud$1.call(DockerCloud.java:290)
        at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

I also notice that there's no longer an option to select the registry URI when configuring the UI:

missing repository uri

What else can I check?

Thanks.

ndeloof commented 6 years ago

So for reason I can't explain yet you don't get authentication string produced by docker-commons API for selected credentials. I can for sure introduce a null check t avoid this ugly NPE but this will just postpone failure to some registry authentication failure.

Can you please check your jenkins log on a fresh start for plugin initialisation errors ? Can you please also use the script console JENKINS_URL/script to run this health check script:

import jenkins.authentication.tokens.api.AuthenticationTokenSource
println Jenkins.getInstance().getExtensionList(AuthenticationTokenSource.class)
ben-gineer commented 6 years ago

I don't see any plugin initialisation errors:

Running from: /usr/share/jenkins/jenkins.war
Oct 19, 2017 10:45:06 PM Main deleteWinstoneTempContents
WARNING: Failed to delete the temporary Winstone file /tmp/winstone/jenkins.war
Oct 19, 2017 10:45:07 PM org.eclipse.jetty.util.log.Log initialized
INFO: Logging initialized @3353ms to org.eclipse.jetty.util.log.JavaUtilLog
Oct 19, 2017 10:45:07 PM winstone.Logger logInternal
INFO: Beginning extraction from war file
Oct 19, 2017 10:45:07 PM org.eclipse.jetty.server.Server doStart
INFO: jetty-9.4.z-SNAPSHOT
Oct 19, 2017 10:45:13 PM org.eclipse.jetty.webapp.StandardDescriptorProcessor visitServlet
INFO: NO JSP Support for /jenkins, did not find org.eclipse.jetty.jsp.JettyJspServlet
Oct 19, 2017 10:45:13 PM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: DefaultSessionIdManager workerName=node0
Oct 19, 2017 10:45:13 PM org.eclipse.jetty.server.session.DefaultSessionIdManager doStart
INFO: No SessionScavenger set, using defaults
Oct 19, 2017 10:45:13 PM org.eclipse.jetty.server.session.HouseKeeper startScavenging
INFO: Scavenging every 660000ms
Jenkins home directory: /opt/jenkins-data found at: EnvVars.masterEnvVars.get("JENKINS_HOME")
Oct 19, 2017 10:45:18 PM org.eclipse.jetty.server.handler.ContextHandler doStart
INFO: Started w.@71075444{/jenkins,file:///var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/war}
Oct 19, 2017 10:45:19 PM org.eclipse.jetty.server.AbstractConnector doStart
INFO: Started ServerConnector@72c8e7b{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
Oct 19, 2017 10:45:19 PM org.eclipse.jetty.server.Server doStart
INFO: Started @15384ms
Oct 19, 2017 10:45:19 PM winstone.Logger logInternal
INFO: Winstone Servlet Engine v4.0 running: controlPort=disabled
Oct 19, 2017 10:45:20 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started initialization
Oct 19, 2017 10:45:32 PM hudson.ClassicPluginStrategy createClassJarFromWebInfClasses
WARNING: Created /opt/jenkins-data/plugins/rebuild/WEB-INF/lib/classes.jar; update plugin to a version created with a newer harness
Oct 19, 2017 10:46:04 PM jenkins.InitReactorRunner$1 onAttained
INFO: Listed all plugins
Oct 19, 2017 10:46:09 PM hudson.plugins.ansicolor.PluginImpl start
INFO: AnsiColor: eliminating boring output (https://github.com/jenkinsci/ansicolor-plugin)
Oct 19, 2017 10:46:11 PM com.nirima.jenkins.RepositoryPlugin start
INFO: Expanding /opt/jenkins-data/plugins/repository/WEB-INF/lib/repository.jarinto /opt/jenkins-data/repositoryPlugin
Oct 19, 2017 10:46:27 PM jenkins.InitReactorRunner$1 onAttained
INFO: Prepared all plugins
Oct 19, 2017 10:46:36 PM jenkins.InitReactorRunner$1 onAttained
INFO: Started all plugins
Oct 19, 2017 10:46:36 PM jenkins.InitReactorRunner$1 onAttained
INFO: Augmented all extensions

Result of script:

[com.cloudbees.jenkins.plugins.amazonecr.AmazonECSRegistryTokenSource@9ebdff2, org.jenkinsci.plugins.docker.commons.impl.ServerKeyMaterialFactoryFromDockerCredentials@615bfe1c, org.jenkinsci.plugins.docker.commons.impl.UsernamePasswordDockerRegistryTokenSource@2009c1ab]
ben-gineer commented 6 years ago

Latest 1.0.2 build seems to resolve this. Thanks

ndeloof commented 6 years ago

✌️