search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.35k stars 1.99k forks source link

Using com.azure.Azure-Identity for MSI based authentication causes linkage errors in jackson.* libraries. #29102

Closed apoorvaguru09 closed 2 years ago

apoorvaguru09 commented 2 years ago

Describe the bug While using Azure-Identity in Scala code to be hosted in Azure HDI cluster, the application throws an error: java.lang.LinkageError: Package versions: jackson-annotations=2.4.4.4.1.8.29, jackson-core=2.4.4.4.1.8.29, jackson-databind=2.4.4.4.1.8.29, jackson-dataformat-xml=unknown, jackson-datatype-jsr310=unknown, com.shaded.azure-core=unknown, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot Even though the correct Jackson.* versions have been pinned to 2.13.2 in the POM xml, the older version: 2.4.4.. is being loaded it seems.

Exception or Stack Trace

22/05/26 09:45:46 ERROR JacksonVersion [Driver]: Version '2.4.4.4.1.8.29' of package 'jackson-annotations' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/05/26 09:45:46 ERROR JacksonVersion [Driver]: Version '2.4.4.4.1.8.29' of package 'jackson-core' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/05/26 09:45:46 ERROR JacksonVersion [Driver]: Version '2.4.4.4.1.8.29' of package 'jackson-databind' is not supported (older than earliest supported version - `2.10.0`), please upgrade.
22/05/26 09:45:46 INFO JacksonVersion [Driver]: Package versions: jackson-annotations=2.4.4.4.1.8.29, jackson-core=2.4.4.4.1.8.29, jackson-databind=2.4.4.4.1.8.29, jackson-dataformat-xml=unknown, jackson-datatype-jsr310=unknown, com.shaded.azure-core=unknown, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
22/05/26 09:45:46 ERROR ObjectMapperShim [Driver]: Package versions: jackson-annotations=2.4.4.4.1.8.29, jackson-core=2.4.4.4.1.8.29, jackson-databind=2.4.4.4.1.8.29, jackson-dataformat-xml=unknown, jackson-datatype-jsr310=unknown, com.shaded.azure-core=unknown, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
22/05/26 09:45:46 ERROR ApplicationMaster [Driver]: User class threw exception: java.lang.LinkageError: Package versions: jackson-annotations=2.4.4.4.1.8.29, jackson-core=2.4.4.4.1.8.29, jackson-databind=2.4.4.4.1.8.29, jackson-dataformat-xml=unknown, jackson-datatype-jsr310=unknown, com.shaded.azure-core=unknown, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
java.lang.LinkageError: Package versions: jackson-annotations=2.4.4.4.1.8.29, jackson-core=2.4.4.4.1.8.29, jackson-databind=2.4.4.4.1.8.29, jackson-dataformat-xml=unknown, jackson-datatype-jsr310=unknown, com.shaded.azure-core=unknown, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
    at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:79)
    at com.azure.core.util.serializer.JacksonAdapter.<init>(JacksonAdapter.java:82)
    at com.azure.core.util.serializer.JacksonAdapter.<init>(JacksonAdapter.java:59)
    at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.<clinit>(JacksonAdapter.java:114)
    at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:123)
    at com.azure.identity.implementation.IdentityClient.<clinit>(IdentityClient.java:104)
    at com.azure.identity.implementation.IdentityClientBuilder.build(IdentityClientBuilder.java:158)
    at com.azure.identity.ManagedIdentityCredential.<init>(ManagedIdentityCredential.java:88)
    at com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:156)
    at com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:150)
    at com.microsoft.mssales.processing.driver.PipelineDriver$.run(PipelineDriver.scala:256)
    at com.microsoft.mssales.processing.driver.PipelineDriver$.main(PipelineDriver.scala:249)
    at com.microsoft.mssales.processing.driver.PipelineDriver.main(PipelineDriver.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684)
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.xml.XmlMapper.coercionConfigDefaults()Lcom/fasterxml/jackson/databind/cfg/MutableCoercionConfig;
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:176)
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:145)
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:127)
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.builder(XmlMapper.java:218)
    at com.azure.core.implementation.jackson.ObjectMapperFactory.createXmlMapper(ObjectMapperFactory.java:89)
    at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:76)
    ... 17 more
22/05/26 09:45:46 INFO ApplicationMaster [Driver]: Final app status: FAILED, exitCode: 13, (reason: User class threw exception: java.lang.LinkageError: Package versions: jackson-annotations=2.4.4.4.1.8.29, jackson-core=2.4.4.4.1.8.29, jackson-databind=2.4.4.4.1.8.29, jackson-dataformat-xml=unknown, jackson-datatype-jsr310=unknown, com.shaded.azure-core=unknown, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
    at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:79)
    at com.azure.core.util.serializer.JacksonAdapter.<init>(JacksonAdapter.java:82)
    at com.azure.core.util.serializer.JacksonAdapter.<init>(JacksonAdapter.java:59)
    at com.azure.core.util.serializer.JacksonAdapter$SerializerAdapterHolder.<clinit>(JacksonAdapter.java:114)
    at com.azure.core.util.serializer.JacksonAdapter.createDefaultSerializerAdapter(JacksonAdapter.java:123)
    at com.azure.identity.implementation.IdentityClient.<clinit>(IdentityClient.java:104)
    at com.azure.identity.implementation.IdentityClientBuilder.build(IdentityClientBuilder.java:158)
    at com.azure.identity.ManagedIdentityCredential.<init>(ManagedIdentityCredential.java:88)
    at com.azure.identity.DefaultAzureCredentialBuilder.getCredentialsChain(DefaultAzureCredentialBuilder.java:156)
    at com.azure.identity.DefaultAzureCredentialBuilder.build(DefaultAzureCredentialBuilder.java:150)
    at com.microsoft.mssales.processing.driver.PipelineDriver$.run(PipelineDriver.scala:256)
    at com.microsoft.mssales.processing.driver.PipelineDriver$.main(PipelineDriver.scala:249)
    at com.microsoft.mssales.processing.driver.PipelineDriver.main(PipelineDriver.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684)
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.dataformat.xml.XmlMapper.coercionConfigDefaults()Lcom/fasterxml/jackson/databind/cfg/MutableCoercionConfig;
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:176)
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:145)
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.<init>(XmlMapper.java:127)
    at com.fasterxml.jackson.dataformat.xml.XmlMapper.builder(XmlMapper.java:218)
    at com.azure.core.implementation.jackson.ObjectMapperFactory.createXmlMapper(ObjectMapperFactory.java:89)
    at com.azure.core.implementation.jackson.ObjectMapperShim.createXmlMapper(ObjectMapperShim.java:76)
    ... 17 more

To Reproduce Steps to reproduce the behavior:

1. Add the dependencies:
<dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-identity</artifactId>
            <version>1.5.1</version>
</dependency>
<dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-security-keyvault-secrets</artifactId>
            <version>4.4.2</version>
</dependency>

  1. Scala code:

    val defaultCred = new DefaultAzureCredentialBuilder().managedIdentityClientId(<**client id of the managed identity.**>).build()
println("Created default creds")
val secretClient = new SecretClientBuilder()
  .vaultUrl("https://<**Your keyvault**>.azure.net/")
  .credential(defaultCred)
  .buildClient();
println("Created SecretClient")

println("Trying to get the value of keyvault key val.")

val secret = secretClient.getSecret("**Your keyvault key**")
println("Retrieved secret with name \"%s\" and value \"%s\"%n", secret.getName, secret.getValue)
  2. Create the package using 'mvn clean package -DskipTests=true'
  3. submit spark job for this package on Azure HDI cluster: Spark 2.4 (HDI 4.0)

Code Snippet Code snippet: Provided in above step#2

Expected behavior A clear and concise description of what you expected to happen. Expectation was to successfully run through the code to try fetch the keyvault key or in worst case : give authorization error due to MSI not having permission,

Screenshots image

Setup (please complete the following information):

-  -<dependency>
-             <groupId>com.azure</groupId>
-             <artifactId>azure-identity</artifactId>
-             <version>1.5.1</version>
- </dependency>
- <dependency>
-             <groupId>com.azure</groupId>
-             <artifactId>azure-security-keyvault-secrets</artifactId>
-             <version>4.4.2</version>
- </dependency>
- <dependency>
-             <groupId>com.fasterxml.jackson.core</groupId>
-             <artifactId>jackson-annotations</artifactId>
-             <version>2.13.3</version>
- </dependency>
- <dependency>
-             <groupId>com.fasterxml.jackson.core</groupId>
-             <artifactId>jackson-core</artifactId>
-             <version>2.13.3</version>
- </dependency>
- <dependency>
-             <groupId>com.fasterxml.jackson.core</groupId>
-             <artifactId>jackson-databind</artifactId>
-             <version>2.13.3</version>
- </dependency>
- <dependency>
-             <groupId>com.fasterxml.jackson.module</groupId>
-             <artifactId>jackson-module-scala_2.11</artifactId>
-             <version>2.13.3</version>
- </dependency>
-  <dependency>
-             <groupId>com.fasterxml.jackson.dataformat</groupId>
-             <artifactId>jackson-dataformat-xml</artifactId>
-             <version>2.13.3</version>
- </dependency>
- <dependency>
-             <groupId>com.fasterxml.jackson.datatype</groupId>
-             <artifactId>jackson-datatype-jsr310</artifactId>
-             <version>2.13.3</version>
- </dependency>

If you suspect a dependency version mismatch (e.g. you see NoClassDefFoundError, NoSuchMethodError or similar), please check out Troubleshoot dependency version conflict article first. If it doesn't provide solution for the problem, please provide:

Additional context Add any other context about the problem here.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

joshfree commented 2 years ago

Hi @apoorvaguru09 thank you for reaching out to us via this GitHub issue; @billwert could you please follow up with @apoorvaguru09?

billwert commented 2 years ago

@alzimmermsft @lmolkova Hey friends - can you help us out here? I suspect this is an issue with ambient state in Spark but I'm not sure how to go about diagnosing or fixing it. @apoorvaguru09 has tried making a fat JAR which seems to have not helped.

apoorvaguru09 commented 2 years ago

Thanks Bill for looking into it, as per suggestion I also tried using the azure-sdk BOM but getting same error. Let me know I can share another Sample Helloworld code in case you want to repro?

billwert commented 2 years ago

@apoorvaguru09 and I worked offline to get the Jackson libraries properly shaded for the Spark environment.

apoorvaguru09 commented 2 years ago

Thanks a lot Bill!