search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.35k stars 1.99k forks source link

[BUG] Jackson issues with Azure SDK when run on Azure Synapse Analytics #25734

Closed purijatin closed 8 months ago

purijatin commented 2 years ago

Describe the bug

We have a spark application that runs on Azure Synapse Spark Pools. Synapse brings a list of dependencies with it, which are by default available on class-path at runtime: https://docs.microsoft.com/en-us/azure/synapse-analytics/spark/Apache-Spark-3-runtime

In the list is Jackson 2.10.

Our project uses PlayJson 2.9.2 that relies on Jackson 2.10. And latest version of Azure SDK (that brings in Jackson 2.12). When run against the Synapse Spark 3 runtime, we get the following exception.

If we remove the azure sdk dependency, then all the code works fine.

Exception or Stack Trace

When run on Synapse, we get the following exception:


21/11/29 17:38:00 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.LinkageError: Package versions: jackson-annotations=2.10.0, jackson-core=2.10.0, jackson-databind=2.10.0, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.12.5, azure-core=1.19.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
    at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:73)
    at com.azure.core.util.serializer.JacksonAdapter.<init>(JacksonAdapter.java:81)
    at com.azure.core.util.serializer.JacksonAdapter.<init>(JacksonAdapter.java:58)
    at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.<clinit>(JacksonAdapter.java:113)
    at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:122)
    at com.azure.identity.implementation.IdentityClient.<clinit>(IdentityClient.java:100)
    at com.azure.identity.implementation.IdentityClientBuilder.build(IdentityClientBuilder.java:139)
    at com.azure.identity.ManagedIdentityCredential.<init>(ManagedIdentityCredential.java:70)
    at com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:129)
    at com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:123)com.microsoft.aiops.anomaly.events.AnomalyEventGridSender.$anonfun$sendEvents$1$adapted(AnomalyEventGridSender.scala:25)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.sendEvents(xxxxxx.scala:25)
    at scala.collection.Iterator.foreach(Iterator.scala:941)
    at scala.collection.Iterator.foreach$(Iterator.scala:941)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
    at scala.collection.IterableLike.foreach(IterableLike.scala:74)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
    at xxxxxx.xxxxxx.xxxxxx.xxxxxx.runner.xxxxxx.xxxxxx(xxxxxx.scala:82)
    at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx(xxxxxx.scala:61)
    at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx.$anonfun$start$2(xxxxxx.scala:39)
    at scala.collection.TraversableLike$WithFilter.$anonfun$map$2(TraversableLike.scala:827)
    at scala.collection.Iterator.foreach(Iterator.scala:941)
    at scala.collection.Iterator.foreach$(Iterator.scala:941)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
    at scala.collection.IterableLike.foreach(IterableLike.scala:74)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
    at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:826)
    at xxxxxx.xxxxxx.xxxxxx.xxxxxx.x.xxxxx.xxxxxx.start(xxxxxx.scala:36)
    at xxxxxx.xxxxxx.xxxxxx.xxxxxx.xxxxxx$.main(xxxxxx.scala:29)
    at xxxxxx.xxxxxx.aiops.xxxxxx.xxxxxx.main(xxxxxx.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732)
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.xml.XmlMapper.coercionConfigDefaults()Lcom/fasterxml/jackson/databind/cfg/MutableCoercionConfig;
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:176)
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:145)
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:127)
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.builder(XmlMapper.java:218)
    at com.azure.core.implementation.jackson.ObjectMapperFactory.createXmlMapper(ObjectMapperFactory.java:84)
    at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:70)
    ... 45 more

To Reproduce Run a synapse spark job with azure-messaging-eventgrid and azure-storage-file-datalake dependency

Code Snippet

Expected behavior

Screenshots tree-2.txt Attached is the dependency tree.

Note: As mentioned above, synapse introduces its own set of dependencies at runtime. And we have no control over it.

Setup (please complete the following information):

purijatin commented 2 years ago

Here is another such exception with Azure SDK and Synapse Spark Pool. 


[info] Cause: com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.10.0 requires Jackson Databind version >= 2.10.0 and < 2.11.0
[info] at com.fasterxml.jackson.module.scala.JacksonModule.setupModule(JacksonModule.scala:61)
[info] at com.fasterxml.jackson.module.scala.JacksonModule.setupModule$(JacksonModule.scala:46)
[info] at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
[info] at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:835)
[info] at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala:82)
[info] at org.apache.spark.rdd.RDDOperationScope$.(RDDOperationScope.scala)
[info] at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
[info] at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:132)
[info] at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:131)

Suggestion

Azure SDK should not have Jackson dependencies. But instead use something else for its Json needs. Jackson doesn't have a good reputation of being compatible across versions. And these kinds of problems will keep popping up. Ultimately affecting the usage of Azure SDK.

joshfree commented 2 years ago

Azure SDK should not have Jackson dependencies. But instead use something else for its Json needs. Jackson doesn't have a good reputation of being compatible across versions. And these kinds of problems will keep popping up. Ultimately affecting the usage of Azure SDK.

Thanks for filing this, @purijatin. @lmolkova is working to reduce our dependency on Jackson.

lmolkova commented 2 years ago

@purijatin thanks for reporting it! We're working on a better story for JAckson, in the meantime, can you please try to specify 2.10 version for all of the Jackson packages?

Based on the error message you posted, there is a mismatch Package versions: jackson-annotations=2.10.0, jackson-core=2.10.0, jackson-databind=2.10.0, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.12.5

So, you'd need to explicitly pin 2.10 version for jackson-dataformat-xml and jackson-datatype-jsr310.

purijatin commented 2 years ago

Hi @lmolkova

As such, I don't have any direct dependency on Jackson. The project has a dependency on spark (which gets in 2.10) series. And azure sdk which gets in 2.12 series.

Do you mean, I should add exclusion for jackson while adding a dependency for azure-sdk? So that only 2.10 remains on the classpath. Will azure-sdk magically work on 2.10?

hasan505 commented 2 years ago

Hi @lmolkova

I am also facing similar issue. As you mentioned in earlier comment I tried with exact versions which are suggested in exception but still issue not resolved. I have excluded the jackson azure-core and explicitly added them. Below is the exception

Exception in thread "main" java.lang.LinkageError: Package versions: jackson-annotations=2.9.0, jackson-core=2.9.8, jackson-databind=2.9.8, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.9.9, azure-core=1.21.0 at com.azure.core.implementation.jackson.ObjectMapperShim.createHeaderMapper(ObjectMapperShim.java:129) at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:75) at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:53) at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:111) at com.azure.storage.blob.implementation.util.ModelHelper.(ModelHelper.java:53) at com.azure.storage.blob.BlobUrlParts.parse(BlobUrlParts.java:371) at com.azure.storage.blob.BlobServiceClientBuilder.endpoint(BlobServiceClientBuilder.java:147) at com.zwayam.storage.AzureStorageProvider.getBlobContainerClient(AzureStorageProvider.java:183)

lmolkova commented 2 years ago

@purijatin yes, please add an explicit dependency on 2.10 for all Jackson packages jackson-annotations, jackson-core, jackson-databind, jackson-dataformat-xml, jackson-datatype-jsr310. And please consider using the latest patch - 2.10.5.

Azure SDK is compatible with jackson 2.10+

lmolkova commented 2 years ago

@hasan505 Based on the log you provided, you're using mixture of versions: Package versions: jackson-annotations=2.9.0, jackson-core=2.9.8, jackson-databind=2.9.8, jackson-dataformat-xml=2.12.5, jackson-datatype-jsr310=2.9.9,.
Make sure you're pinning versions for all packages and note that Azure SDK is compatible with Jackson 2.10+.

hasan505 commented 2 years ago

@lmolkova Yes, It's an issue with different versions of jackson dependencies. I have removed them and added 2.12.5 which is working fine now. Thank you for the assistance.

purijatin commented 2 years ago

@purijatin yes, please add an explicit dependency on 2.10 for all Jackson packages jackson-annotations, jackson-core, jackson-databind, jackson-dataformat-xml, jackson-datatype-jsr310. And please consider using the latest patch - 2.10.5.

Azure SDK is compatible with jackson 2.10+

Thanks. Let me try it out.

Suggestion: If azure-sdk supports 2.10+, then it shouldn't have a dependency for 2.12 series. By doing for 2.12, it unintentionally evicts the older Jackson versions that other libraries may introduce. Giving rise to conflicts like above.

If azure-sdk had relied on Jackson 2.10, then even if other libraries had relied on 2.11/2.12, maven/gradle/sbt would have evicted the 2.10 version by default and used the latest version. And things would have worked (because azure-sdk works with 2.11/2.12 on classpath). It is not perfect but may work for simple cases by default.

I again feel that libraries like azure-sdk should not rely on Jackson. Maybe use something else for its Json needs. These kinds of issues are never-ending with Jackson and it has remained like this for years. It is not worth the pain IMHO. Though I don't think it would be easy now as it would break the user's code for example in BinaryData.fromObject.

Thanks again for all the great work and for helping proactively with these issues.

lmolkova commented 2 years ago

thanks for the ideas! If we pin Jackson to 2.10 we'd expose users to some known bugs and vulnerabilities. Doing so now would also be breaking. While we're considering reducing our dependency on Jackson libraries, this is a big investment, we track this work here https://github.com/Azure/azure-sdk-for-java/issues/24480.

limotley commented 2 years ago

I'm having a similar issue when trying to use the Azure SDK (azure-data-tables package) on Synapse with Spark 3.1, although it seems to be the inverse where an Azure SDK package is not compatible with Jackson 2.10. The code runs fine in local tests, but when I try to submit it to the Synapse cluster, I get this:

java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.xml.XmlMapper.coercionConfigDefaults()Lcom/fasterxml/jackson/databind/cfg/MutableCoercionConfig; at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:176) at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:145) at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:127) at com.fasterxml.jackson.dataformat.xml.XmlMapper.builder(XmlMapper.java:218) at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:140) at com.azure.data.tables.implementation.TablesJacksonSerializer.(TablesJacksonSerializer.java:27) at com.azure.data.tables.TableClientBuilder.(TableClientBuilder.java:54) at .....

this is with azure-data-tables 12.1.2. Based on the comments above, I thought all Azure SDK packages were compatible with 2.10?

lmolkova commented 2 years ago

@limotley please go through the Troubleshooting guide. Most likely you have a mismatch between versions of different Jackson packages. The guide mentions all Jackson packages that Azure SDKs use and most likely you'd need to pin each of them to a specific version supported by Synapse.

If this does not help, please create a new issue and provide the information requested in the issue template. Thanks!

limotley commented 2 years ago

@lmolkova thanks for the tip! I had the Jackson packages pinned, but their version number wasn't set to the same one as Spark's. Working now

nishanthvasu commented 2 years ago

I also face the same issue. Can anyone help me out in this?

2022-03-23 15:46:02.077 19310-19337/com.downloadfile E/AndroidRuntime: FATAL EXCEPTION: pool-2-thread-1 Process: com.downloadfile, PID: 19310 java.lang.LinkageError: Package versions: jackson-annotations=0.0, jackson-core=0.0, jackson-databind=0.0, jackson-dataformat-xml=0.0, jackson-datatype-jsr310=0.0, azure-core=1.26.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:73) at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:81) at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:58) at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.(JacksonAdapter.java:113) at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.access$000(JacksonAdapter.java:109) at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:122) at com.azure.storage.blob.implementation.util.ModelHelper.(ModelHelper.java:63) at com.azure.storage.blob.implementation.util.ModelHelper.determineAuthorityIsIpStyle(ModelHelper.java:82) at com.azure.storage.blob.BlobUrlParts.parse(BlobUrlParts.java:371) at com.azure.storage.blob.BlobServiceClientBuilder.endpoint(BlobServiceClientBuilder.java:171) at com.azure.storage.blob.BlobServiceClientBuilder.connectionString(BlobServiceClientBuilder.java:328) at com.downloadfile.FileDownloadTask.download(FileDownloadTask.java:85) at com.downloadfile.FileDownloadTask.run(FileDownloadTask.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:923) Caused by: java.lang.NoSuchMethodError: No static method newFactory(Ljava/lang/String;Ljava/lang/ClassLoader;)Ljavax/xml/stream/XMLInputFactory; in class Ljavax/xml/stream/XMLInputFactory; or its super classes (declaration of 'javax.xml.stream.XMLInputFactory' appears in /data/data/com.downloadfile/code_cache/.overlay/base.apk/classes3.dex) at com.fasterxml.jackson.dataformat.xml.XmlFactory.(XmlFactory.java:115) at com.fasterxml.jackson.dataformat.xml.XmlFactory.(XmlFactory.java:101) at com.fasterxml.jackson.dataformat.xml.XmlFactory.(XmlFactory.java:85) at com.fasterxml.jackson.dataformat.xml.XmlMapper.(XmlMapper.java:127) at com.fasterxml.jackson.dataformat.xml.XmlMapper.builder(XmlMapper.java:218) at com.azure.core.implementation.jackson.ObjectMapperFactory.createXmlMapper(ObjectMapperFactory.java:85) at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:70) at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:81)  at com.azure.core.util.serializer.JacksonAdapter.(JacksonAdapter.java:58)  at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.(JacksonAdapter.java:113)  at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.access$000(JacksonAdapter.java:109)  at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:122)  at com.azure.storage.blob.implementation.util.ModelHelper.(ModelHelper.java:63)  at com.azure.storage.blob.implementation.util.ModelHelper.determineAuthorityIsIpStyle(ModelHelper.java:82)  at com.azure.storage.blob.BlobUrlParts.parse(BlobUrlParts.java:371)  at com.azure.storage.blob.BlobServiceClientBuilder.endpoint(BlobServiceClientBuilder.java:171)  at com.azure.storage.blob.BlobServiceClientBuilder.connectionString(BlobServiceClientBuilder.java:328)  at com.downloadfile.FileDownloadTask.download(FileDownloadTask.java:85)  at com.downloadfile.FileDownloadTask.run(FileDownloadTask.java:49)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)  at java.lang.Thread.run(Thread.java:923) 

jmakuc commented 2 years ago

It also happens when you use Kafka and Azure Datalake SDK together. My move was to use a previous version of the azure dependency to avoid conflicts, accepting the vulnerabilities @lmolkova says.

jbauerrfid commented 2 years ago

Had similar issues with Spring Boot and Azure Cognitive Search using the wrong Jackson version. Considering that we currently have 10+ Spring applications using Spring Boot 2.5.2 with carefully selected dependencies I don't want to update all of them just because of Microsoft...

The way out was to identity the Search version using the wanted Jackson version like this:

com.azure/azure-search-documents: 11.4.0
    |
    +-> com.azure/azure-core: 1.17.0
        |
    +-> com.fasterxml.jackson-*/*: 2.12.3
        |
org.springframework.boot/spring-boot-starter-webflux: 2.5.2

Maybe not the best solution, because I cannot update Azure Search easily, but at least I got rid of the version conflicts,

shrutisarda23 commented 1 year ago

Add all the required dependencies explicitly in your pom.xml and it will resolve. for example:

com.fasterxml.jackson.dataformat jackson-dataformat-xml 2.8.9 com.fasterxml.jackson.datatype jackson-datatype-jsr310 2.8.9 com.fasterxml.jackson.core jackson-core 2.8.9 com.fasterxml.jackson.core jackson-databind 2.8.9
github-actions[bot] commented 8 months ago

Hi @purijatin, we deeply appreciate your input into this project. Regrettably, this issue has remained inactive for over 2 years, leading us to the decision to close it. We've implemented this policy to maintain the relevance of our issue queue and facilitate easier navigation for new contributors. If you still believe this topic requires attention, please feel free to create a new issue, referencing this one. Thank you for your understanding and ongoing support.