search

xenon-middleware / xenon-cli

Perform files and jobs operations with Xenon library from command line
http://nlesc.github.io/Xenon/
Apache License 2.0
2 stars 3 forks source link

Jersey conflicts #53

Open sverhoeven opened 6 years ago

sverhoeven commented 6 years ago

In branch https://github.com/NLeSC/xenon-cli/tree/prep-xenon230 I want to use the Xenon v2.3.0, but the integration tests are failing with 

java.lang.NoSuchMethodError: javax.ws.rs.core.MultivaluedMap.addAll(Ljava/lang/Object;[Ljava/lang/Object;)V
    at org.glassfish.jersey.client.ClientRequest.accept(ClientRequest.java:327)
    at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:229)
    at org.glassfish.jersey.client.JerseyWebTarget.request(JerseyWebTarget.java:59)
    at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:1249)
    at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:1224)
    at com.spotify.docker.client.DefaultDockerClient.pull(DefaultDockerClient.java:1218)
    at com.github.geowarin.junit.DockerRule.<init>(DockerRule.java:60)
    at com.github.geowarin.junit.DockerRuleBuilder.build(DockerRuleBuilder.java:87)
    at nl.esciencecenter.xenon.cli.SftpTest.<clinit>(SftpTest.java:31)
    at sun.misc.Unsafe.ensureClassInitialized(Native Method)
    at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
    at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
    at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
    at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
    at java.lang.reflect.Field.get(Field.java:393)
    at org.junit.runners.model.FrameworkField.get(FrameworkField.java:73)
    at org.junit.runners.model.TestClass.getAnnotatedFieldValues(TestClass.java:230)
    at org.junit.runners.ParentRunner.classRules(ParentRunner.java:255)
    at org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:244)
    at org.junit.runners.ParentRunner.classBlock(ParentRunner.java:194)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:362)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
    at com.sun.proxy.$Proxy1.processTestClass(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:108)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:146)
    at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:128)
    at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:404)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
    at java.lang.Thread.run(Thread.java:748)

The new dependencies of Xenon are conflicting the docker rule used in the integration tests.

sverhoeven commented 6 years ago

In https://github.com/spotify/docker-client/blob/master/pom.xml it depends on jersey 2.22.2. In ./gradlew dependencies jersey 1.19 is a dependency of several hadoop libraries.

The MultivaluedMap.addAll was introduced in JAX-RS 2 which jersey2 implements.

jmaassen commented 6 years ago

Any suggestions on fixing this ? Maybe hadoop also works with the 2.22.2 version of jersey ?

sverhoeven commented 6 years ago

I think one of Hadoop dependencies pulls in the JAX-RS 1 version of MultivaluedMap. I could exclude that dependency, but this will probably break hadoop.

sverhoeven commented 6 years ago

We might be able to shade the javax/ws/rs/ classes in jersey-core-1.9.jar using http://imperceptiblethoughts.com/shadow/#relocating_packages so hadoop uses the shaded javax/ws/rs/ and spotify uses the ones that ship with java.

@jmaassen do you any experience shading?

jmaassen commented 6 years ago

No experience with shading....

What we used to do in an earlier incarnation of Xenon is use separate class loaders for the different adaptors.

Since I had similar issues with the GridFTP and SSH adaptor clashing over a dependency, this may be the time to seriously start looking for a solution

jmaassen commented 6 years ago

I'll compare some potential solutions

sverhoeven commented 6 years ago

The 3.0.0 hadoop client shadows it's Jersey deps.

Adding integrationTestCompile 'org.apache.hadoop:hadoop-client:3.0.0' as dependency and using Xenon 2.4.0 gives no errors.

The ./gradlew dependencies contains org.apache.hadoop:hadoop-common:2.8.1 -> 3.0.0 and com.sun.jersey:jersey-json:1.19 so this probably works because the classpath order is changed.