Cannot connect using docker for mac with unix socket in 8.9.2

johnflavin commented 6 years ago

Description

After updating my library to version 8.9.2, I cannot connect to docker with a unix socket. The errors I get suggest something is wrong with the new jnr version from #918.

How to reproduce

final DockerClient client = DefaultDockerClient.builder()
                .uri("unix:///var/run/docker.sock")
                .build();
client.ping()

What do you expect

The client will connect to docker via the unix socket. The ping will return successfully.

What happened instead

The ping fails with an exception. See stack trace below.

Software:

Spotify's docker-client version: 8.9.2

docker for mac

Version 17.11.0-ce-rc3-mac38 (20152)
Channel: edge
a091b4ad0a

docker version


Client:
Version:      17.11.0-ce-rc3
API version:  1.34
Go version:   go1.8.4
Git commit:   5b4af4f
Built:        Wed Nov  8 03:03:18 2017
OS/Arch:      darwin/amd64

Server: Version: 17.11.0-ce-rc3 API version: 1.34 (minimum version 1.12) Go version: go1.8.5 Git commit: 5b4af4f Built: Wed Nov 8 03:09:46 2017 OS/Arch: linux/amd64 Experimental: true

- macOS 10.13.1

## Full backtrace

```text
08:34:18,227  WARN DockerControlApiTest:116 - Could not connect to docker.
com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.lang.NoSuchMethodError: jnr.ffi.Platform.getStandardCLibraryName()Ljava/lang/String;
    at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2699)
    at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2630)
    at com.spotify.docker.client.DefaultDockerClient.ping(DefaultDockerClient.java:502)
    at org.nrg.containers.api.DockerControlApiTest.canConnectToDocker(DockerControlApiTest.java:114)
    at org.nrg.containers.api.DockerControlApiTest.testPullImage(DockerControlApiTest.java:186)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
    at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
    at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
    at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.lang.NoSuchMethodError: jnr.ffi.Platform.getStandardCLibraryName()Ljava/lang/String;
    at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
    at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2628)
    ... 55 more
Caused by: javax.ws.rs.ProcessingException: java.lang.NoSuchMethodError: jnr.ffi.Platform.getStandardCLibraryName()Ljava/lang/String;
    at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:202)
    at org.glassfish.jersey.client.ClientRuntime.access$400(ClientRuntime.java:79)
    at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:182)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
    at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    ... 3 more
Caused by: java.lang.NoSuchMethodError: jnr.ffi.Platform.getStandardCLibraryName()Ljava/lang/String;
    at jnr.unixsocket.Native.<clinit>(Native.java:48)
    at jnr.unixsocket.UnixSocketChannel.<init>(UnixSocketChannel.java:101)
    at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60)
    at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:69)
    at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:44)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:118)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
    at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435)
    at org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50)
    at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
    at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487)
    at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:178)
    ... 12 more

johnflavin commented 6 years ago

To verify this wasn't an issue with the old versions of these dependencies being cached, I removed all the jnr dependencies from my gradle and maven caches. When I recompiled my project, it downloaded these versions:

Download https://jcenter.bintray.com/com/github/jnr/jnr-unixsocket/0.18/jnr-unixsocket-0.18.jar
Download https://jcenter.bintray.com/com/github/jnr/jnr-ffi/2.1.4/jnr-ffi-2.1.4.jar
Download https://jcenter.bintray.com/com/github/jnr/jnr-constants/0.9.8/jnr-constants-0.9.8.jar
Download https://jcenter.bintray.com/com/github/jnr/jnr-enxio/0.16/jnr-enxio-0.16.jar
Download https://jcenter.bintray.com/com/github/jnr/jnr-posix/3.0.35/jnr-posix-3.0.35.jar
Download https://jcenter.bintray.com/com/github/jnr/jffi/1.2.15/jffi-1.2.15.jar
Download https://jcenter.bintray.com/com/github/jnr/jffi/1.2.15/jffi-1.2.15-native.jar
Download https://jcenter.bintray.com/com/github/jnr/jnr-x86asm/1.0.2/jnr-x86asm-1.0.2.jar

I get the same error.

davidxia commented 6 years ago

@johnflavin Thanks for reporting. I was able to run this code successfully.

    final DockerClient client = DefaultDockerClient.builder()
        .uri("unix:///var/run/docker.sock")
        .build();
    client.ping();
    System.out.println(client.version());

I deleted my cached jnr jars before hand. I ran on ubuntu 14.04 trusty with docker version

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:06:06 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:06:06 2017
 OS/Arch:      linux/amd64
 Experimental: false

dxia@guc3-heliosjenkins-b-49ey:~/helios$ ls -ail /var/run/docker.sock
2996509702 srw-rw---- 1 root docker 0 Nov 13 06:20 /var/run/docker.sock

The system user I used to run the tests is in the docker group.

On my mac I use docker-machine and connect on a TCP socket. Are you using docker for mac?

johnflavin commented 6 years ago

Yes, this occurs using docker for mac and connecting over a unix socket. (And, side note, docker for mac cannot be configured to use TCP. If you try to set the hosts option in the settings json, it conflicts with the --hosts option hard-coded into the docker command line options in the xhyve VM. See docker/for-mac#1595. Not particularly relevant, but I wanted to kvetch about it.)

When I create a docker-machine VM and connect over TCP, everything runs fine.

But I would think that isn't surprising. The jnr changes were about the code for unix socket connections. Nothing about the TCP connections should be impacted.

johnflavin commented 6 years ago

I've updated the issue title and description to explicitly include docker for mac details.

davidxia commented 6 years ago

Thanks for the update. My first test case I wrote about above was on ubuntu with unix socket. Let me try docker for mac now.

johnflavin commented 6 years ago

I think I've figured out that this is a problem with one of my other dependencies packaging an older version of the jnr libraries. This isn't a docker-client problem, it's just a me problem.

davidxia commented 6 years ago

phew, thanks. I was worried for a second. For the record, the shaded docker-client jar works with unix socket on DFM running docker 17.09.

spotify / docker-client