[BUG] java.lang.NoSuchMethodError: io.netty.handler.ssl.SslProvider.isAlpnSupported for Azure Storage Queue SDK

rodfsouza commented 4 years ago

Describe the bug I'm currently using Spark 2.4.4, my project is a consumer taking data from Kafka, doing an aggregation and send to Azure Queue however, whenever trying to connect to Azure Storage Queue by using SSL throws the exception below and no data is send.

Exception or Stack Trace java.lang.NoSuchMethodError: io.netty.handler.ssl.SslProvider.isAlpnSupported(Lio/netty/handler/ssl/SslProvider;)Z at reactor.netty.http.client.HttpClientSecure.(HttpClientSecure.java:79) at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:301) at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57) at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.resubscribe(FluxRetryPredicate.java:124) at reactor.core.publisher.MonoRetryPredicate.subscribeOrReturn(MonoRetryPredicate.java:51) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:326) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) at reactor.core.publisher.MonoDelaySubscription.accept(MonoDelaySubscription.java:52) at reactor.core.publisher.MonoDelaySubscription.accept(MonoDelaySubscription.java:33) at reactor.core.publisher.FluxDelaySubscription$DelaySubscriptionOtherSubscriber.onNext(FluxDelaySubscription.java:123) at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

To Reproduce Create a Maven project, and use the following Pom.xml as below:

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

4.0.0

<groupId>com.microsoft</groupId>
<artifactId>CoreRankerSparkStructuredStreaming</artifactId>
<version>1.5</version>

<properties>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <encoding>UTF-8</encoding>
    <spark.version>2.4.4</spark.version>
    <scala.version.major>2.11</scala.version.major>
    <scala.version.minor>8</scala.version.minor>
    <scala.version>${scala.version.major}.${scala.version.minor}</scala.version>
</properties>

<repositories>
    <repository>
        <id>UEQ</id>
        <url>https://msasg.pkgs.visualstudio.com/Bing_Ads/_packaging/UEQ/maven/v1</url>
        <releases>
            <enabled>true</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>com.microsoft.bingads.ueq</groupId>
        <artifactId>ueq-schema</artifactId>
        <version>1.6.2</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.version.major}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
        <exclusions>
            <exclusion>  <!-- declare the exclusion here -->
                <groupId>io.netty</groupId>
                <artifactId>netty</artifactId>
            </exclusion>
            <exclusion>  <!-- declare the exclusion here -->
                <groupId>io.netty</groupId>
                <artifactId>netty-all</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.version.major}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>2.3.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql-kafka-0-10_${scala.version.major}</artifactId>
        <version>${spark.version}</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/com.azure/azure-storage-queue -->
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-storage-queue</artifactId>
        <version>12.5.2</version>
    </dependency>

    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-core</artifactId>
        <version>1.5.1</version> <!-- {x-version-update;com.azure:azure-core;dependency} -->
    </dependency>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-storage-common</artifactId>
        <version>12.7.0</version> <!-- {x-version-update;com.azure:azure-storage-common;current} -->
    </dependency>

    <!-- Azure Event Hub -->
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-messaging-eventhubs</artifactId>
        <version>5.1.1</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core -->
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>2.11.1</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/io.netty/netty-all -->
    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-all</artifactId>
        <version>4.1.49.Final</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/io.netty/netty-handler -->

    <!-- https://mvnrepository.com/artifact/io.projectreactor.netty/reactor-netty -->

    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter-api</artifactId>
        <version>5.7.0-M1</version>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <resources>
        <resource>
            <directory>src/main</directory>
            <includes>
                <include>resources/distinct_domain_names.csv</include>
            </includes>
        </resource>
    </resources>
    <plugins>

        <!--            <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-dependency-plugin</artifactId>
                        <version>2.6</version>
                        <executions>
                            <execution>
                                <id>unpack-dependencies</id>
                                <phase>package</phase>
                                <goals>
                                    <goal>unpack-dependencies</goal>
                                </goals>
                                <configuration>
                                    <excludeScope>system</excludeScope>
                                    <excludes>META-INF/*.SF,META-INF/*.DSA,META-INF/*.RSA</excludes>
                                    <excludeGroupIds>junit,org.mockito,org.hamcrest</excludeGroupIds>
                                    <outputDirectory>${project.build.directory}/classes</outputDirectory>
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>-->

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.1</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
                <useIncrementalCompilation>false</useIncrementalCompilation>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.4</version>
            <configuration>
                <!-- put your configurations here -->
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <filters>
                            <filter>
                                <artifact>*:*</artifact>
                                <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                </excludes>
                            </filter>
                        </filters>
                    </configuration>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <artifactId>maven-resources-plugin</artifactId>
            <version>3.0.0</version>
            <executions>
                <execution>
                    <id>copy-resources</id>
                    <!-- here the phase you need -->
                    <phase>validate</phase>
                    <goals>
                        <goal>copy-resources</goal>
                    </goals>
                    <configuration>
                        <outputDirectory>${basedir}/target</outputDirectory>
                        <resources>
                            <resource>
                                <directory>src/main/resources</directory>
                                <filtering>true</filtering>
                            </resource>
                        </resources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Code Snippet public void send(List<Tuple2<String, java.sql.Timestamp>> events) { try { String queueName = props.getProperty(AZURE_QUEUE_NAME_KEY); QueueClient client = AzureQueueFactory.getQueueClient(CONN_STRING, queueName, 10, 60);

        for(Tuple2<String, java.sql.Timestamp> event : events) {
            Timestamp time = new Timestamp(event._2.getTime(), System.currentTimeMillis());
            Message msg = new Message(UUID.randomUUID().toString(), event._1, COUNT_STR, time);
            String msgStr = mapper.writeValueAsString(msg);
            LOGGER.info("Sending message: {}", msgStr);
            client.sendMessage(msgStr);
        }
    } catch (Exception ex) {
        LOGGER.error("Exception while processing the events", ex);
    }
}

Expected behavior I would expect the connection to be properly open and the data be sent with no issues.

Screenshots If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

OS: Windows
IDE : IntelliJ
Version of the Library used: 12.5.2

Additional context mvn dependency:tree

[INFO] com.microsoft:CoreRankerSparkStructuredStreaming:jar:1.5 [INFO] +- com.microsoft.bingads.ueq:ueq-schema:jar:1.6.2:compile [INFO] +- org.apache.spark:spark-core_2.11:jar:2.4.4:provided [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.8:provided [INFO] | +- org.apache.avro:avro:jar:1.8.2:provided [INFO] | | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided [INFO] | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided [INFO] | | +- org.apache.commons:commons-compress:jar:1.8.1:provided [INFO] | | - org.tukaani:xz:jar:1.5:provided [INFO] | +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.2:provided [INFO] | | +- org.apache.avro:avro-ipc:jar:1.8.2:provided [INFO] | | - commons-codec:commons-codec:jar:1.9:provided [INFO] | +- com.twitter:chill_2.11:jar:0.9.3:provided [INFO] | | - com.esotericsoftware:kryo-shaded:jar:4.0.2:provided [INFO] | | +- com.esotericsoftware:minlog:jar:1.3.0:provided [INFO] | | - org.objenesis:objenesis:jar:2.5.1:provided [INFO] | +- com.twitter:chill-java:jar:0.9.3:provided [INFO] | +- org.apache.xbean:xbean-asm6-shaded:jar:4.8:provided [INFO] | +- org.apache.hadoop:hadoop-client:jar:2.6.5:provided [INFO] | | +- org.apache.hadoop:hadoop-common:jar:2.6.5:provided [INFO] | | | +- commons-cli:commons-cli:jar:1.2:provided [INFO] | | | +- xmlenc:xmlenc:jar:0.52:provided [INFO] | | | +- commons-httpclient:commons-httpclient:jar:3.1:provided [INFO] | | | +- commons-io:commons-io:jar:2.4:provided [INFO] | | | +- commons-collections:commons-collections:jar:3.2.2:provided [INFO] | | | +- commons-configuration:commons-configuration:jar:1.6:provided [INFO] | | | | - commons-digester:commons-digester:jar:1.8:provided [INFO] | | | | - commons-beanutils:commons-beanutils:jar:1.7.0:provided [INFO] | | | +- com.google.code.gson:gson:jar:2.2.4:provided [INFO] | | | +- org.apache.hadoop:hadoop-auth:jar:2.6.5:provided [INFO] | | | | +- org.apache.httpcomponents:httpclient:jar:4.2.5:provided [INFO] | | | | | - org.apache.httpcomponents:httpcore:jar:4.2.4:provided [INFO] | | | | - org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:provided [INFO] | | | | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:provided [INFO] | | | | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:provided [INFO] | | | | - org.apache.directory.api:api-util:jar:1.0.0-M20:provided [INFO] | | | +- org.apache.curator:curator-client:jar:2.6.0:provided [INFO] | | | - org.htrace:htrace-core:jar:3.0.4:provided [INFO] | | +- org.apache.hadoop:hadoop-hdfs:jar:2.6.5:provided [INFO] | | | +- org.mortbay.jetty:jetty-util:jar:6.1.26:provided [INFO] | | | - xerces:xercesImpl:jar:2.9.1:provided [INFO] | | | - xml-apis:xml-apis:jar:1.3.04:provided [INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.5:provided [INFO] | | | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.5:provided [INFO] | | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.5:provided [INFO] | | | | - org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.5:provided [INFO] | | | - org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.5:provided [INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.5:provided [INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.5:provided [INFO] | | | - org.apache.hadoop:hadoop-yarn-common:jar:2.6.5:provided [INFO] | | | +- javax.xml.bind:jaxb-api:jar:2.2.2:provided [INFO] | | | | - javax.xml.stream:stax-api:jar:1.0-2:provided [INFO] | | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:provided [INFO] | | | - org.codehaus.jackson:jackson-xc:jar:1.9.13:provided [INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.5:provided [INFO] | | - org.apache.hadoop:hadoop-annotations:jar:2.6.5:provided [INFO] | +- org.apache.spark:spark-launcher_2.11:jar:2.4.4:provided [INFO] | +- org.apache.spark:spark-kvstore_2.11:jar:2.4.4:provided [INFO] | | +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided [INFO] | | - com.fasterxml.jackson.core:jackson-annotations:jar:2.6.7:compile [INFO] | +- org.apache.spark:spark-network-common_2.11:jar:2.4.4:provided [INFO] | +- org.apache.spark:spark-network-shuffle_2.11:jar:2.4.4:provided [INFO] | +- org.apache.spark:spark-unsafe_2.11:jar:2.4.4:provided [INFO] | +- javax.activation:activation:jar:1.1.1:provided [INFO] | +- org.apache.curator:curator-recipes:jar:2.6.0:provided [INFO] | | +- org.apache.curator:curator-framework:jar:2.6.0:provided [INFO] | | - com.google.guava:guava:jar:16.0.1:provided [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.6:provided [INFO] | +- javax.servlet:javax.servlet-api:jar:3.1.0:provided [INFO] | +- org.apache.commons:commons-lang3:jar:3.5:provided [INFO] | +- org.apache.commons:commons-math3:jar:3.4.1:provided [INFO] | +- com.google.code.findbugs:jsr305:jar:1.3.9:provided [INFO] | +- org.slf4j:slf4j-api:jar:1.7.16:compile [INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.16:provided [INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.16:provided [INFO] | +- log4j:log4j:jar:1.2.17:provided [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.16:provided [INFO] | +- com.ning:compress-lzf:jar:1.0.3:provided [INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.7.3:compile [INFO] | +- org.lz4:lz4-java:jar:1.4.0:compile [INFO] | +- com.github.luben:zstd-jni:jar:1.3.2-2:compile [INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.7.45:provided [INFO] | | - org.roaringbitmap:shims:jar:0.7.45:provided [INFO] | +- commons-net:commons-net:jar:3.1:provided [INFO] | +- org.scala-lang:scala-library:jar:2.11.12:compile [INFO] | +- org.json4s:json4s-jackson_2.11:jar:3.5.3:provided [INFO] | | - org.json4s:json4s-core_2.11:jar:3.5.3:provided [INFO] | | +- org.json4s:json4s-ast_2.11:jar:3.5.3:provided [INFO] | | +- org.json4s:json4s-scalap_2.11:jar:3.5.3:provided [INFO] | | - org.scala-lang.modules:scala-xml_2.11:jar:1.0.6:provided [INFO] | +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:provided [INFO] | | +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided [INFO] | | +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:provided [INFO] | | | +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:provided [INFO] | | | - org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:provided [INFO] | | +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:provided [INFO] | | - org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:provided [INFO] | | - org.javassist:javassist:jar:3.18.1-GA:provided [INFO] | +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:provided [INFO] | | +- javax.annotation:javax.annotation-api:jar:1.2:provided [INFO] | | +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:provided [INFO] | | - org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided [INFO] | +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:provided [INFO] | | +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:provided [INFO] | | - javax.validation:validation-api:jar:1.1.0.Final:provided [INFO] | +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:provided [INFO] | +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:provided [INFO] | +- com.clearspring.analytics:stream:jar:2.7.0:provided [INFO] | +- io.dropwizard.metrics:metrics-core:jar:3.1.5:provided [INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:3.1.5:provided [INFO] | +- io.dropwizard.metrics:metrics-json:jar:3.1.5:provided [INFO] | +- io.dropwizard.metrics:metrics-graphite:jar:3.1.5:provided [INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.7.1:compile [INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.7.1:provided [INFO] | | +- org.scala-lang:scala-reflect:jar:2.11.8:provided [INFO] | | - com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.7.9:provided [INFO] | +- org.apache.ivy:ivy:jar:2.4.0:provided [INFO] | +- oro:oro:jar:2.0.8:provided [INFO] | +- net.razorvine:pyrolite:jar:4.13:provided [INFO] | +- net.sf.py4j:py4j:jar:0.10.7:provided [INFO] | +- org.apache.spark:spark-tags_2.11:jar:2.4.4:compile [INFO] | +- org.apache.commons:commons-crypto:jar:1.0.0:provided [INFO] | - org.spark-project.spark:unused:jar:1.0.0:compile [INFO] +- org.apache.spark:spark-sql_2.11:jar:2.4.4:provided [INFO] | +- com.univocity:univocity-parsers:jar:2.7.3:provided [INFO] | +- org.apache.spark:spark-sketch_2.11:jar:2.4.4:provided [INFO] | +- org.apache.spark:spark-catalyst_2.11:jar:2.4.4:provided [INFO] | | +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.1.0:provided [INFO] | | +- org.codehaus.janino:janino:jar:3.0.9:provided [INFO] | | +- org.codehaus.janino:commons-compiler:jar:3.0.9:provided [INFO] | | - org.antlr:antlr4-runtime:jar:4.7:provided [INFO] | +- org.apache.orc:orc-core:jar:nohive:1.5.5:provided [INFO] | | +- org.apache.orc:orc-shims:jar:1.5.5:provided [INFO] | | +- com.google.protobuf:protobuf-java:jar:2.5.0:provided [INFO] | | +- commons-lang:commons-lang:jar:2.6:provided [INFO] | | - io.airlift:aircompressor:jar:0.10:provided [INFO] | +- org.apache.orc:orc-mapreduce:jar:nohive:1.5.5:provided [INFO] | +- org.apache.parquet:parquet-column:jar:1.10.1:provided [INFO] | | +- org.apache.parquet:parquet-common:jar:1.10.1:provided [INFO] | | - org.apache.parquet:parquet-encoding:jar:1.10.1:provided [INFO] | +- org.apache.parquet:parquet-hadoop:jar:1.10.1:provided [INFO] | | +- org.apache.parquet:parquet-format:jar:2.4.0:provided [INFO] | | - org.apache.parquet:parquet-jackson:jar:1.10.1:provided [INFO] | - org.apache.arrow:arrow-vector:jar:0.10.0:provided [INFO] | +- org.apache.arrow:arrow-format:jar:0.10.0:provided [INFO] | +- org.apache.arrow:arrow-memory:jar:0.10.0:provided [INFO] | +- joda-time:joda-time:jar:2.9.9:provided [INFO] | +- com.carrotsearch:hppc:jar:0.7.2:provided [INFO] | - com.vlkan:flatbuffers:jar:1.2.0-3f79e055:provided [INFO] +- org.apache.kafka:kafka-clients:jar:2.3.1:compile [INFO] +- org.apache.spark:spark-sql-kafka-0-10_2.11:jar:2.4.4:compile [INFO] +- com.azure:azure-storage-queue:jar:12.5.2:compile [INFO] +- com.azure:azure-core:jar:1.5.1:compile [INFO] | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.10.1:compile [INFO] | +- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.10.1:compile [INFO] | | +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.1:compile [INFO] | | | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile [INFO] | | | - jakarta.activation:jakarta.activation-api:jar:1.2.1:compile [INFO] | | +- org.codehaus.woodstox:stax2-api:jar:4.2:compile [INFO] | | - com.fasterxml.woodstox:woodstox-core:jar:6.0.2:compile [INFO] | +- io.projectreactor:reactor-core:jar:3.3.5.RELEASE:compile [INFO] | | - org.reactivestreams:reactive-streams:jar:1.0.3:compile [INFO] | - io.netty:netty-tcnative-boringssl-static:jar:2.0.29.Final:compile [INFO] +- com.azure:azure-storage-common:jar:12.7.0:compile [INFO] | - com.azure:azure-core-http-netty:jar:1.5.2:compile [INFO] | +- io.netty:netty-handler:jar:4.1.49.Final:compile [INFO] | | +- io.netty:netty-common:jar:4.1.49.Final:compile [INFO] | | +- io.netty:netty-resolver:jar:4.1.49.Final:compile [INFO] | | +- io.netty:netty-transport:jar:4.1.49.Final:compile [INFO] | | - io.netty:netty-codec:jar:4.1.49.Final:compile [INFO] | +- io.netty:netty-handler-proxy:jar:4.1.49.Final:compile [INFO] | | - io.netty:netty-codec-socks:jar:4.1.49.Final:compile [INFO] | +- io.netty:netty-buffer:jar:4.1.49.Final:compile [INFO] | +- io.netty:netty-codec-http:jar:4.1.49.Final:compile [INFO] | +- io.netty:netty-codec-http2:jar:4.1.49.Final:compile [INFO] | +- io.netty:netty-transport-native-unix-common:jar:4.1.49.Final:compile [INFO] | +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.49.Final:compile [INFO] | - io.projectreactor.netty:reactor-netty:jar:0.9.7.RELEASE:compile [INFO] +- com.azure:azure-messaging-eventhubs:jar:5.1.1:compile [INFO] | - com.azure:azure-core-amqp:jar:1.2.0:compile [INFO] | +- com.microsoft.azure:qpid-proton-j-extensions:jar:1.2.3:compile [INFO] | - org.apache.qpid:proton-j:jar:0.33.4:compile [INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.11.1:compile [INFO] +- io.netty:netty-all:jar:4.1.49.Final:compile [INFO] - org.junit.jupiter:junit-jupiter-api:jar:5.7.0-M1:test [INFO] +- org.apiguardian:apiguardian-api:jar:1.1.0:test [INFO] +- org.opentest4j:opentest4j:jar:1.2.0:test [INFO] - org.junit.platform:junit-platform-commons:jar:1.7.0-M1:test

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x ] Bug Description Added
[ x] Repro Steps Added
[ x] Setup information Added

joshfree commented 4 years ago

@anuchandy / @alzimmermsft could you please assist?

joshfree commented 4 years ago

https://github.com/Azure/azure-sdk-for-java/wiki/Frequently-Asked-Questions#dependency-conflict

anuchandy commented 4 years ago

Resolved offline. @rodfsouza, I'm closing this now that the app is working on spark.

bsikander commented 4 years ago

Just writing my solution for others.

In my case, at runtime, I had a conflict with netty versions. My azure-storage-file-datalake dependency was bringing in netty version 4.1.51.Final whereas the netty version required by spark 2.3.0 was 4.1.17.Final. When I was deploying my job, netty version 4.1.17.Final was given precedence which doesn't have the function io.netty.handler.ssl.SslProvider.isAlpnSupported.

Since, I had no control over the dependencies of the Spark cluster. The only option left for me was to shade the io.netty in my fat jar. Here is what I added in my sbt file

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("io.netty.**" -> "shadenetty.@1").inAll
)

This allows both conflicting dependencies to live together since the package name is changed to shadenetty..

zzgzy commented 3 years ago

@anuchandy Would you mind talking a bit more about the solution? Is it related to shaded jar? In my case, I added below to my pom to shade the jar.

  <relocation>
      <pattern>io.netty</pattern>
      <shadedPattern>com.microsoft.shaded.io.netty</shadedPattern>
      <includes>
          <include>io.netty.**</include>
      </includes>
  </relocation>

Thank you.

Azure / azure-sdk-for-java

[BUG] java.lang.NoSuchMethodError: io.netty.handler.ssl.SslProvider.isAlpnSupported for Azure Storage Queue SDK #12534