jenkinsci / docker-plugin

Jenkins cloud plugin that uses Docker
https://plugins.jenkins.io/docker-plugin/
MIT License
490 stars 320 forks source link

Can't connect to docker using TLS and Docker credentials #371

Closed samrocketman closed 7 years ago

samrocketman commented 8 years ago

Server configuration

I had to dig for it but found the docker cloud authentication configuration instructions https://github.com/jenkinsci/docker-plugin/issues/245#issuecomment-158012362.

Jenkins specs

I can't get the docker plugin to properly connect. I click the test connection button and it throws an error. When I test with curl it succeeds with my client certificates.

cd /path/docker
#successful auth test
curl --cert ./cert.pem --key ./key.pem --cacert ./ca.pem https://127.0.0.1:443/info
#successfully denied access
curl -k https://127.0.0.1:443/info

Here's a stack trace from the web UI when I use the Test Connection button in the docker cloud settings of the configure page.

Error in SSL Configuration

com.github.dockerjava.api.DockerClientException: Error in SSL Configuration
    at com.github.dockerjava.jaxrs.DockerCmdExecFactoryImpl.init(DockerCmdExecFactoryImpl.java:140)
    at com.github.dockerjava.core.DockerClientImpl.withDockerCmdExecFactory(DockerClientImpl.java:131)
    at com.github.dockerjava.core.DockerClientBuilder.build(DockerClientBuilder.java:73)
    at com.nirima.jenkins.plugins.docker.client.ClientBuilderForPlugin.build(ClientBuilderForPlugin.java:65)
    at com.nirima.jenkins.plugins.docker.DockerCloud$DescriptorImpl.doTestConnection(DockerCloud.java:615)
    at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.github.dockerjava.api.DockerClientException: org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to org.bouncycastle.openssl.PEMKeyPair
    at com.github.dockerjava.core.LocalDirectorySSLConfig.getSSLContext(LocalDirectorySSLConfig.java:57)
    at com.github.dockerjava.jaxrs.DockerCmdExecFactoryImpl.init(DockerCmdExecFactoryImpl.java:138)
    ... 61 more
Caused by: java.lang.ClassCastException: org.bouncycastle.asn1.pkcs.PrivateKeyInfo cannot be cast to org.bouncycastle.openssl.PEMKeyPair
    at com.github.dockerjava.core.CertificateUtils.loadPrivateKey(CertificateUtils.java:127)
    at com.github.dockerjava.core.CertificateUtils.createKeyStore(CertificateUtils.java:45)
    at com.github.dockerjava.core.LocalDirectorySSLConfig.getSSLContext(LocalDirectorySSLConfig.java:50)
    ... 62 more

Please note my SSL certs are x509 ASCII PEM format. Any idea what could be going wrong?

Workaround

This section is an edit of the original issue. I noted a workaround below.

samrocketman commented 8 years ago

@magnayn I'm able to reproduce the stack trace from the Jenkins script console.

//variables
String serverUrl = "https://127.0.0.1:443"
String credentialsId = "606ed7a0-0cab-4c76-bd57-a9f4aea9a692"
String version = "1.8"
Integer readTimeout = 0
Integer connectTimeout = 0

//import com.nirima.jenkins.plugins.docker.DockerCloud$DescriptorImpl
//def x = Jenkins.instance.getExtensionList(com.nirima.jenkins.plugins.docker.DockerCloud$DescriptorImpl)[0]
//println x.metaClass.methods*.name.sort().unique()
//x.doTestConnection(serverUrl, credentialsId, version, readTimeout, connectTimeout)

import com.github.dockerjava.core.DockerClientConfig
import com.nirima.jenkins.plugins.docker.client.DockerCmdExecConfig
import com.github.dockerjava.api.DockerClient
import com.github.dockerjava.api.model.Version
import com.nirima.jenkins.plugins.docker.client.ClientConfigBuilderForPlugin
import com.nirima.jenkins.plugins.docker.client.DockerCmdExecConfigBuilderForPlugin
import com.nirima.jenkins.plugins.docker.client.ClientBuilderForPlugin

final DockerClientConfig clientConfig = ClientConfigBuilderForPlugin.dockerClientConfig()
    .forServer(serverUrl, version)
    .withCredentials(credentialsId)
    .build()

final DockerCmdExecConfig execConfig = DockerCmdExecConfigBuilderForPlugin.builder()
    .withReadTimeout(readTimeout)
    .withConnectTimeout(connectTimeout)
    .build()

DockerClient dc = ClientBuilderForPlugin.builder()
    .withDockerClientConfig(clientConfig)
    .withDockerCmdExecConfig(execConfig)
    .build()

Version verResult = dc.versionCmd().exec()
"Version = " + verResult.getVersion()

It's basically an excerpt of doTestConnection(). The exception is called when the following is executed:

DockerClient dc = ClientBuilderForPlugin.builder()
    .withDockerClientConfig(clientConfig)
    .withDockerCmdExecConfig(execConfig)
    .build()
samrocketman commented 8 years ago

Upgraded to Docker version 1.9.1, build a34a1d5. curl still works and Jenkins config does not. It doesn't appear to reach a step of connecting to docker. It seems to fail on setup before a connection is ever made to docker.

samrocketman commented 8 years ago

Here's some example docker certificates if you want to test what I am doing. Please note, even when I remove the PEM info from the certs it still doesn't work and consistently fails with the same error.

samrocketman commented 8 years ago

The root of the exception occurs here. I've checked my 4096-bit private key and it seems OK.

$ openssl rsa -in key.pem -check -noout
RSA key ok
samrocketman commented 8 years ago

It FINALLY worked because I had to change how the private key was presented.

mv key.pem key.bak
openssl rsa -in key.bak -text > key.pem

Apparently, it only takes the private key successfully if you give it an extended -text format. The wiki needs to be updated with how to use TLS auth. I'll get around to it if nobody does.

samrocketman commented 8 years ago

Filed a bug report with upstream library.

samrocketman commented 8 years ago

Since https://github.com/docker-java/docker-java/issues/441 is closed and the maintainer says they can't reproduce with their library using a different plugin then it narrows it to this plugin.

samrocketman commented 8 years ago

As noted in https://github.com/docker-java/docker-java/issues/441 there seems to be an open PR which could potentially fix this issue. I'm looking forward to it.

See the fix at https://github.com/docker-java/docker-java/pull/502

jquast commented 8 years ago

Hey @samrocketman I just wanted to thank you for discovering the problem and workaround, helped me through my day! Thank you!

samrocketman commented 8 years ago

Awesome @jquast :D glad my experience could help you. I went ahead and updated the original issue so the workaround is pointed out right away.

ndeloof commented 7 years ago

Docker plugin 0.17 now uses docker-commons for credentials management, which should avoid such issues making clearer definition about the expected keys format.