search

jenkinsci / docker-plugin

Jenkins cloud plugin that uses Docker
https://plugins.jenkins.io/docker-plugin/
MIT License
490 stars 319 forks source link

Plugin Doesn't Respect ~/.docker/config.json "credHelpers" settings #803

Open craigwatson opened 4 years ago

craigwatson commented 4 years ago

We are currently using the Docker plugin to run builds Docker images on a remote server.

Previously, we have been hosting our Docker images on a private Artifactory repository.

Recently, we moved our images to a private Google Container Repository (GCR) repo.

Our Jenkins master and Docker hosts are running on GCE, so they have been set up with the credHelper settings in /home/jenkins/.docker/config.json:

$ cat /home/jenkins/docker/config.json
{
  "credHelpers": {
    "gcr.io": "gcloud",
    "us.gcr.io": "gcloud",
    "eu.gcr.io": "gcloud",
    "asia.gcr.io": "gcloud",
    "staging-k8s.gcr.io": "gcloud",
    "marketplace.gcr.io": "gcloud"
  }
}

This allows the Jenkins user to pull images via the CLI, and authentication is handled via the underlying gcloud library, using the GCE instance's machine account.

However, the Jenkins Docker plugin requires a Service Account JSON key and cannot - it seems - use the GCE machine account, either via a Jenkins Credential or by specifying none and allowing the Java library to fall back to the native configuration. In each case, the below exception is shown on the Jenkins master's "Configure Clouds" page:

com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication"}
    at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:109)
  at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)
  at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
  at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
  at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
  at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
  at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
  at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
  at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
  at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
  at java.lang.Thread.run(Thread.java:748)

This has been tested and verified by changing the image to a publicly-accessible GCR image - the exception does not occur here because the unauthenticated client can access it.

pjdarton commented 4 years ago

You are correct that the docker-plugin doesn't pay any attention to any files on the Jenkins master (aside from the Jenkins configuration files that configures Jenkins itself) or on the remote agents. The docker-plugin does not use the docker command-line client; it uses docker-java (a docker client written in Java) to access the docker API of any docker hosts it talks to; this doesn't pay any attention to any files on the filesystem either.

I'm not familiar with this /home/jenkins/docker/config.json file you've listed, or the /home/jenkins/.docker/config.json file you mention; neither are relevant to the docker-plugin as that's not how it's configured (as you've found out) ... but it's not meant to be looking at them either (by design, Jenkins functionality is configured through Jenkins, not through other files "scattered around" the filesystem - the idea is that Jenkins users don't need access to the "native" filesystem that Jenkins is hosted on). i.e. it's not meant to "fall back to the native configuration" (as far as I am aware).

I'm not familiar with this gcloud thing either, so I can't offer much advice there either.

IME, if a docker resource can work like a standard docker host and/or standard docker registry then it'll work fine with the docker-plugin. If it isn't a standard docker host/registry (and can't pretend to be one) then your options are more limited.

If you're running into authentication issues then your options are limited to just what the docker-plugin's configuration pages provide you with - you can provide credentials to Jenkins and you can instruct the docker-plugin to use those when talking to a docker host and/or a docker registry (the latter is akin to doing docker login at the command-line). I'm not familiar with the term "Service Account JSON key" - as far as I am aware, credentials are a username + password ... although it's not unusual for the "password" to actually be some form of encoded key (specific to the authentication layer of whatever you're talking to, e.g. an Artifactory API key if the registry is actually Artifactory).

TL;DR: You need to figure out how to access "GCR" without using a (specific to the docker CLI) "credHelper" because this plugin doesn't use the docker CLI at all.

craigwatson commented 4 years ago

@pjdarton Thanks for the response :)

Regarding the configuration files, that makes perfect sense - I was under the impression that the Java plugin would somehow read the local configuration if it was present and use that for its defaults.

Within Google Cloud, you have the option of using a "service account" to authenticate to Google services - as far as I'm aware, this is based on time-limited OAuth credentials, rather than a static username/password. The end result is that rather than providing a username/password, you "inherit" the credentials from the OS, and authenticate the "service account", rather than the "user".

The Google Cloud SDK (packaged as the gcloud binary, but also exposed as various APIs in most languages) exposes this natively (e.g. if you want to push a file to Google Cloud Storage, you don't need to pass any username/password - you can use gsutil cp natively and the OS inherits the authentication of the service account).

The SDK also exposes various "credential helpers" to provide usernames and passwords for services like Google Cloud Source Repositories and Google Container Repositories, which rely on traditional credential pairs. For example, a gcloud helper will supply git with HTTP Basic Auth credentials (which are programmatically generated) for a Google Cloud Source Repository. The same exists for docker - from 18.03 and up - full details are here.

The alternative to this helper is to supply a JSON key (and in the process circumvent the "inherit from the OS" method) for the service account. This does actually work with the Jenkins Docker plugin as it ends up being a standard username/password combination, however it means we need to set up and manage an entirely different service account rather than leverage the underlying mechanism provided (and recommended by) Google.

It seems there's an issue open within the docker-java library in order to implement the credsHelper options - I'll keep an eye on that PR and bump this issue when support is added, so that it can be ported into the plugin 👍

https://github.com/docker-java/docker-java/issues/1048

dsielert commented 2 years ago

it seems as though the referenced upstream issue with docker-java is closed @craigwatson . If I'm reading https://github.com/docker-java/docker-java/issues/1048#issuecomment-646451804 correctly it seems you're supposed to be able to pass in a client configuration ?

pjdarton commented 2 years ago

FYI there have been (relatively) recent changes that have updated the version of docker-java present to 3.2.x. That upgrade may unblock this issue ... but I've no idea (and have no time to investigate myself).

So, if anyone is interested in having this feature, please investigate and report back here. I'll clear the "blocked-by-dependency" flag if docker-java 3.2.x allows this and folks can collaborate on getting a PR put together.