Add shaded artifact for Spotify implementation

[nonsleepr]

Resolves #123

[nonsleepr]

@viktortnk Are you publishing only to Maven or to Ivy as well? This PR would create proper pom but would mess up (at least local) ivy.

[nonsleepr]

Ready to be merged. Tested by publishing to Artifactory.

[nonsleepr]

The change publishes a separate artifact com.whisk:docker-testkit-impl-spotify-shaded_2.12:0.9.7 which dependes on shaded uberjar from Spotify ("com.spotify" % "docker-client" % "8.11.5" classifier "shaded"). Alternative would be to produce another uberjar which would include Spotify's uberjar.

[nonsleepr]

@viktortnk Do you think you could merge it?

[viktortnk]

I don't understand the reason for the issue. Can someone else explain why ClassCastException happens. The solution should be excluding spotify dependency from docker-testkit and including shaded one

[nonsleepr]

Okay. Here's an elaborate explanation.

I'm trying to use com.whisk:docker-testkit-scalatest with com.whisk:docker-testkit-impl-spotify to run integration tests of my Spark app. Out of the box, I'm getting the following exception:

libraryDependencies += Seq()
  "com.whisk" %% "docker-testkit-scalatest" % "0.9.7",
  "com.whisk" %% "docker-testkit-impl-spotify" % "0.9.7",
)

com.spotify.docker.client.exceptions.DockerException

com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2812)
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2666)
at com.spotify.docker.client.DefaultDockerClient.listImages(DefaultDockerClient.java:690)
at com.whisk.docker.impl.spotify.SpotifyDockerCommandExecutor$$anonfun$listImages$1.apply(SpotifyDockerCommandExecutor.scala:185)
at com.whisk.docker.impl.spotify.SpotifyDockerCommandExecutor$$anonfun$listImages$1.apply(SpotifyDockerCommandExecutor.scala:188)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
...
Cause: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
at jersey.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2664)
at com.spotify.docker.client.DefaultDockerClient.listImages(DefaultDockerClient.java:690)
at com.whisk.docker.impl.spotify.SpotifyDockerCommandExecutor$$anonfun$listImages$1.apply(SpotifyDockerCommandExecutor.scala:185)
at com.whisk.docker.impl.spotify.SpotifyDockerCommandExecutor$$anonfun$listImages$1.apply(SpotifyDockerCommandExecutor.scala:188)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
...
Cause: javax.ws.rs.ProcessingException: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:202)
at org.glassfish.jersey.client.ClientRuntime.access$400(ClientRuntime.java:79)
at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:182)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:340)
at org.glassfish.jersey.client.ClientRuntime$3.run(ClientRuntime.java:210)
...
Cause: java.lang.IncompatibleClassChangeError: Found interface org.objectweb.asm.ClassVisitor, but class was expected
at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:104)
at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:89)
at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:44)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:325)
at jnr.unixsocket.Native.(Native.java:80)
at jnr.unixsocket.UnixSocketChannel.(UnixSocketChannel.java:101)
at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60)
at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:69)
at com.spotify.docker.client.UnixConnectionSocketFactory.createSocket(UnixConnectionSocketFactory.java:44)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:118)

This problem is discussed and solved by spotify/docker-client#272 and requires using shaded package.

So, the solution is to use shaded docker-client so that it wouldn't conflict with Spark's dependencies.

But if I change my libraryDependencies to look like that:

libraryDependencies ++= {
  ...
  "com.whisk" %% "docker-testkit-scalatest" % "0.9.7",
  "com.whisk" %% "docker-testkit-impl-spotify" % "0.9.7",
    exclude("com.spotify", "docker-client"),
  "com.spotify" % "docker-client" % "8.11.5" classifier "shaded"

}

and try to run my tests, I will get another error.

Here's a simple code snippet to reporduce the problem:

import scala.concurrent.ExecutionContext.Implicits.global
import com.spotify.docker.client.DefaultDockerClient
import com.whisk.docker.impl.spotify.SpotifyDockerCommandExecutor

val client = DefaultDockerClient.fromEnv().build()
val ce = new SpotifyDockerCommandExecutor("localhost", client)
ce.listImages()

Above causes following exception:

java.lang.NoSuchMethodError: com.spotify.docker.client.messages.Image.repoTags()Lcom/google/common/collect/ImmutableList;
    at com.whisk.docker.impl.spotify.SpotifyDockerCommandExecutor$$anonfun$listImages$1$$anonfun$apply$8.apply(SpotifyDockerCommandExecutor.scala:187)
    at com.whisk.docker.impl.spotify.SpotifyDockerCommandExecutor$$anonfun$listImages$1$$anonfun$apply$8.apply(SpotifyDockerCommandExecutor.scala:187)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
    at com.whisk.docker.impl.spotify.SpotifyDockerCommandExecutor$$anonfun$listImages$1.apply(SpotifyDockerCommandExecutor.scala:187)
    at com.whisk.docker.impl.spotify.SpotifyDockerCommandExecutor$$anonfun$listImages$1.apply(SpotifyDockerCommandExecutor.scala:188)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Let's see why:

client.listImages().get(0).repoTags()
//scala> client.listImages().get(0).repoTags()
res1: com.spotify.docker.client.shaded.com.google.common.collect.ImmutableList[String] = [...]

So, docker-it-scala expects com.google.common.collect.ImmutableList but receives com.spotify.docker.client.shaded.com.google.common.collect.ImmutableList. And this is why this library needs shaded version.

[viktortnk]

Thanks for detailed explanation. Makes sense