Dependencies conflict building fat jar with com.microsoft.azure

Hey,

Problem: building a fat jar results in a run time error of NoSuchMethod.

Details:

I'm using the SDK for Scala on Spark integration on HDI. When I build a simple application creating a KeyVaultClient using ApplicationTokenCredentials object from com.microsoft.azure.credentials and run it locally everything works. Building a fat jar with mvn works but shows the following:

com.microsoft.azure:azure-keyvault:jar:1.1.1:compile [INFO] +- com.microsoft.rest:client-runtime:jar:1.6.1:compile [INFO] | +- (com.squareup.okhttp3:okhttp:jar:3.11.0:compile - omitted for conflict with 3.10.0) [INFO] | +- com.squareup.okhttp3:logging-interceptor:jar:3.11.0:compile [INFO] | | - (com.squareup.okhttp3:okhttp:jar:3.11.0:compile - omitted for duplicate) [INFO] | - com.squareup.okhttp3:okhttp-urlconnection:jar:3.11.0:compile [INFO] | - (com.squareup.okhttp3:okhttp:jar:3.11.0:compile - omitted for duplicate) [INFO] +- com.squareup.retrofit2:retrofit:jar:2.4.0:compile [INFO] | - (com.squareup.okhttp3:okhttp:jar:3.10.0:compile - omitted for conflict with 3.11.0) [INFO] +- com.squareup.okhttp3:okhttp:jar:3.10.0:compile [INFO] | - (com.squareup.okio:okio:jar:1.14.0:compile - omitted for duplicate) [INFO] - com.squareup.okio:okio:jar:1.14.0:compile

Deploying the JAR to HDI and running it fails with the following trimmed stacktrace:

18/10/18 09:00:25 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NoSuchMethodError: okio.BufferedSource.readUtf8LineStrict(J)Ljava/lang/String; at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215) at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)

My pom.xml contains spark dependencies and the following:

    <dependency>
        <groupId>com.microsoft.azure</groupId>
        <artifactId>azure</artifactId>
        <version>1.14.0</version>
    </dependency>
    <dependency>
        <groupId>com.microsoft.azure</groupId>
        <artifactId>azure-keyvault</artifactId>
        <version>1.1.1</version>
    </dependency>

At this moment, as a work around I added this to my pom.xml file (before the two above) which forces the downgrade of azure-client that com.microsoft.azure uses (1.6):

    <dependency>
        <groupId>com.microsoft.rest</groupId>
        <artifactId>client-runtime</artifactId>
        <version>1.5.4</version>
    </dependency>

FYI, I think when pulling dependencies in clojure I receive \a similar warning, and suggests a similar solution to yours with the lein build tools:

project.clj:

  :dependencies [[org.clojure/clojure "1.9.0"]
                 ;; https://mvnrepository.com/artifact/com.microsoft.azure/azure-storage
                 [com.microsoft.azure/azure-storage "8.0.0"]
                 [com.microsoft.azure/azure-storage-blob "10.1.0"]
                 [com.microsoft.azure/azure-keyvault-complete "1.1.1" :extension "pom" ]
                 ]

lein deps
WARNING!!! version ranges found for:

[com.microsoft.azure/azure-keyvault-complete "1.1.1" :extension "pom"] -> 
[com.microsoft.azure/azure-keyvault-extensions "1.1.1"] -> 
[com.microsoft.azure/azure-mgmt-resources "1.3.0"] -> 
[com.microsoft.azure/azure-client-runtime "[1.1.0,2.0.0)"]

Consider using [com.microsoft.azure/azure-keyvault-complete "1.1.1" :extension "pom" :exclusions [com.microsoft.azure/azure-client-runtime]].

Azure / azure-keyvault-java

Dependencies conflict building fat jar with com.microsoft.azure #62