search

AbsaOSS / spline

Data Lineage Tracking And Visualization Solution
https://absaoss.github.io/spline/
Apache License 2.0
598 stars 154 forks source link

throwing error on databricks when using the jar built locally #688

Closed uday1409 closed 4 years ago

uday1409 commented 4 years ago

Hi,

I have cloned the code from spark-agent repo and packaged the jar locally without any modifications (used Spark 2.4 bundle), and installed jar on databricks cluster.

As soon as I run any command , its failing with following error. After googling about this, found that it will occur when we use incompatible versions for (io.delta -2.11) and scala (2.12). Can you please check this if any of the dependencies referring to wrong scala version. I think databricks by default has (io.delta -2.11) based on cluster config.

Databricks cluster config: Spark 2.4.5 and Scala 2.11 (using same config, I was able to capture lineage with the already packaged jars provided by you)

com.google.common.util.concurrent.ExecutionError: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.delta.sources.DeltaDataSource could not be instantiated

If I uninstall jar from cluster, all works well.

PS: I was able to capture lineage from the pre built artifacts provided by you in Maven repo. I am facing this error when I packaged the jar locally.

uday1409 commented 4 years ago

On further investigation, I found that when I pointed to version 0.5.1 of agent-core artifact, the locally packaged jar is working fine. I suspect something wrong with agent-core version 0.5.2.

Also, I am planning to modify spark agent -core module, hence I request you to place proper versions of artifacts as the name stands. Appreciate your help !!

wajda commented 4 years ago

Spline agent doesn't have any dependency on io.delta. Make sure you re using a correct Spline agent bundle version. It has to match your Spark version and Scala version.

wajda commented 4 years ago

Also can you try Spline agent 0.5.3, that has been released today?

uday1409 commented 4 years ago

@wajda true that it does not have any dependency on io.delta. But databricks clusters imports io.delta by default same version as its config ( in this case scala 2.11). And Spark Agent Core_2.11 0.5.2 probbaly had scala version 2.12 ( or its dependent package spline-comms had 2.12, it was the case in the parent-pom file at least in rest/comms module code) . Please note Spark Agent Core_2.11 0.5.1 had worked as said earlier.

It would really helpful, if you could place all the dependent artifacts with respective version as their name suggests.

I will try with 0.5.3 as well. I am anyway planning to modify spark agent core module. Hence I request you to place all artifacts with proper versions as their name suggests,

wajda commented 4 years ago

No, it's not possible for 2.11 bundle to depend on anything compiled against Scala 2.12 So there has to be something else that causing the conflict. You can check the full dependency list by running the following command from the corresponding bundle-x.x directory:

mvn dependency:tree

To switch between Scala versions in the project run one of the following for Scala 2.11 or 2.12 respectively:

mvn scala-cross-build:change-version -Pscala-2.11
mvn scala-cross-build:change-version -Pscala-2.12

The dependency tree for bundle-2.4_2.11: (Note the vast majority dependencies are provided, meaning they aren't bundled)

[INFO] Scanning for projects...
[INFO] 
[INFO] --< za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.11 >--
[INFO] Building spark-2.4-spline-agent-bundle_2.11 0.5.3
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ spark-2.4-spline-agent-bundle_2.11 ---
[INFO] za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.11:jar:0.5.3
[INFO] +- za.co.absa.spline.agent.spark:agent-core_2.11:jar:0.5.3:compile
[INFO] |  +- org.scala-lang:scala-compiler:jar:2.11.12:provided
[INFO] |  |  \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:provided
[INFO] |  +- za.co.absa.commons:commons_2.11:jar:0.0.13:compile
[INFO] |  +- org.apache.spark:spark-core_2.11:jar:2.4.2:provided
[INFO] |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.8:provided
[INFO] |  |  +- org.apache.avro:avro:jar:1.8.2:provided
[INFO] |  |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided
[INFO] |  |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided
[INFO] |  |  |  +- org.apache.commons:commons-compress:jar:1.8.1:provided
[INFO] |  |  |  \- org.tukaani:xz:jar:1.5:provided
[INFO] |  |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.2:provided
[INFO] |  |  |  +- org.apache.avro:avro-ipc:jar:1.8.2:provided
[INFO] |  |  |  \- commons-codec:commons-codec:jar:1.9:provided
[INFO] |  |  +- com.twitter:chill_2.11:jar:0.9.3:provided
[INFO] |  |  |  \- com.esotericsoftware:kryo-shaded:jar:4.0.2:provided
[INFO] |  |  |     \- com.esotericsoftware:minlog:jar:1.3.0:provided
[INFO] |  |  +- com.twitter:chill-java:jar:0.9.3:provided
[INFO] |  |  +- org.apache.xbean:xbean-asm6-shaded:jar:4.8:provided
[INFO] |  |  +- org.apache.hadoop:hadoop-client:jar:2.6.5:provided
[INFO] |  |  |  +- org.apache.hadoop:hadoop-common:jar:2.6.5:provided
[INFO] |  |  |  |  +- commons-cli:commons-cli:jar:1.2:provided
[INFO] |  |  |  |  +- xmlenc:xmlenc:jar:0.52:provided
[INFO] |  |  |  |  +- commons-httpclient:commons-httpclient:jar:3.1:provided
[INFO] |  |  |  |  +- commons-io:commons-io:jar:2.4:provided
[INFO] |  |  |  |  +- commons-collections:commons-collections:jar:3.2.2:provided
[INFO] |  |  |  |  +- commons-configuration:commons-configuration:jar:1.6:provided
[INFO] |  |  |  |  |  +- commons-digester:commons-digester:jar:1.8:provided
[INFO] |  |  |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:provided
[INFO] |  |  |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:provided
[INFO] |  |  |  |  +- com.google.code.gson:gson:jar:2.2.4:provided
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-auth:jar:2.6.5:provided
[INFO] |  |  |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.2.5:provided
[INFO] |  |  |  |  |  |  \- org.apache.httpcomponents:httpcore:jar:4.2.4:provided
[INFO] |  |  |  |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:provided
[INFO] |  |  |  |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:provided
[INFO] |  |  |  |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:provided
[INFO] |  |  |  |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:provided
[INFO] |  |  |  |  +- org.apache.curator:curator-client:jar:2.6.0:provided
[INFO] |  |  |  |  \- org.htrace:htrace-core:jar:3.0.4:provided
[INFO] |  |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.6.5:provided
[INFO] |  |  |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:provided
[INFO] |  |  |  |  \- xerces:xercesImpl:jar:2.9.1:provided
[INFO] |  |  |  |     \- xml-apis:xml-apis:jar:1.3.04:provided
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.5:provided
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.5:provided
[INFO] |  |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.5:provided
[INFO] |  |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.5:provided
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.5:provided
[INFO] |  |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.5:provided
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.5:provided
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.5:provided
[INFO] |  |  |  |     +- javax.xml.bind:jaxb-api:jar:2.2.2:provided
[INFO] |  |  |  |     |  \- javax.xml.stream:stax-api:jar:1.0-2:provided
[INFO] |  |  |  |     +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:provided
[INFO] |  |  |  |     \- org.codehaus.jackson:jackson-xc:jar:1.9.13:provided
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.5:provided
[INFO] |  |  |  \- org.apache.hadoop:hadoop-annotations:jar:2.6.5:provided
[INFO] |  |  +- org.apache.spark:spark-launcher_2.11:jar:2.4.2:provided
[INFO] |  |  +- org.apache.spark:spark-kvstore_2.11:jar:2.4.2:provided
[INFO] |  |  |  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-core:jar:2.6.7:provided
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.7:provided
[INFO] |  |  +- org.apache.spark:spark-network-common_2.11:jar:2.4.2:provided
[INFO] |  |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.4.2:provided
[INFO] |  |  +- org.apache.spark:spark-unsafe_2.11:jar:2.4.2:provided
[INFO] |  |  +- javax.activation:activation:jar:1.1.1:provided
[INFO] |  |  +- org.apache.curator:curator-recipes:jar:2.6.0:provided
[INFO] |  |  |  +- org.apache.curator:curator-framework:jar:2.6.0:provided
[INFO] |  |  |  \- com.google.guava:guava:jar:16.0.1:provided
[INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
[INFO] |  |  +- javax.servlet:javax.servlet-api:jar:3.1.0:provided
[INFO] |  |  +- org.apache.commons:commons-lang3:jar:3.5:provided
[INFO] |  |  +- org.apache.commons:commons-math3:jar:3.4.1:provided
[INFO] |  |  +- org.slf4j:slf4j-api:jar:1.7.16:provided
[INFO] |  |  +- org.slf4j:jul-to-slf4j:jar:1.7.16:provided
[INFO] |  |  +- org.slf4j:jcl-over-slf4j:jar:1.7.16:provided
[INFO] |  |  +- log4j:log4j:jar:1.2.17:provided
[INFO] |  |  +- org.slf4j:slf4j-log4j12:jar:1.7.16:provided
[INFO] |  |  +- com.ning:compress-lzf:jar:1.0.3:provided
[INFO] |  |  +- org.xerial.snappy:snappy-java:jar:1.1.7.3:provided
[INFO] |  |  +- org.lz4:lz4-java:jar:1.4.0:provided
[INFO] |  |  +- com.github.luben:zstd-jni:jar:1.3.2-2:provided
[INFO] |  |  +- org.roaringbitmap:RoaringBitmap:jar:0.7.45:provided
[INFO] |  |  |  \- org.roaringbitmap:shims:jar:0.7.45:provided
[INFO] |  |  +- commons-net:commons-net:jar:3.1:provided
[INFO] |  |  +- org.json4s:json4s-jackson_2.11:jar:3.5.3:provided
[INFO] |  |  |  \- org.json4s:json4s-core_2.11:jar:3.5.3:provided
[INFO] |  |  |     +- org.json4s:json4s-ast_2.11:jar:3.5.3:provided
[INFO] |  |  |     \- org.json4s:json4s-scalap_2.11:jar:3.5.3:provided
[INFO] |  |  +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:provided
[INFO] |  |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
[INFO] |  |  |  +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:provided
[INFO] |  |  |  |  +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:provided
[INFO] |  |  |  |  \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:provided
[INFO] |  |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:provided
[INFO] |  |  |  \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:provided
[INFO] |  |  |     \- org.javassist:javassist:jar:3.18.1-GA:provided
[INFO] |  |  +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:provided
[INFO] |  |  |  +- javax.annotation:javax.annotation-api:jar:1.2:provided
[INFO] |  |  |  +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:provided
[INFO] |  |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
[INFO] |  |  +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:provided
[INFO] |  |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:provided
[INFO] |  |  |  \- javax.validation:validation-api:jar:1.1.0.Final:provided
[INFO] |  |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:provided
[INFO] |  |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:provided
[INFO] |  |  +- io.netty:netty-all:jar:4.1.17.Final:provided
[INFO] |  |  +- io.netty:netty:jar:3.9.9.Final:provided
[INFO] |  |  +- com.clearspring.analytics:stream:jar:2.7.0:provided
[INFO] |  |  +- io.dropwizard.metrics:metrics-core:jar:3.1.5:provided
[INFO] |  |  +- io.dropwizard.metrics:metrics-jvm:jar:3.1.5:provided
[INFO] |  |  +- io.dropwizard.metrics:metrics-json:jar:3.1.5:provided
[INFO] |  |  +- io.dropwizard.metrics:metrics-graphite:jar:3.1.5:provided
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.7.1:provided
[INFO] |  |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.7.1:provided
[INFO] |  |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.7.9:provided
[INFO] |  |  +- org.apache.ivy:ivy:jar:2.4.0:provided
[INFO] |  |  +- oro:oro:jar:2.0.8:provided
[INFO] |  |  +- net.razorvine:pyrolite:jar:4.13:provided
[INFO] |  |  +- net.sf.py4j:py4j:jar:0.10.7:provided
[INFO] |  |  +- org.apache.spark:spark-tags_2.11:jar:2.4.2:provided
[INFO] |  |  +- org.apache.commons:commons-crypto:jar:1.0.0:provided
[INFO] |  |  \- org.spark-project.spark:unused:jar:1.0.0:provided
[INFO] |  +- org.apache.spark:spark-sql_2.11:jar:2.4.2:provided
[INFO] |  |  +- com.univocity:univocity-parsers:jar:2.7.3:provided
[INFO] |  |  +- org.apache.spark:spark-sketch_2.11:jar:2.4.2:provided
[INFO] |  |  +- org.apache.spark:spark-catalyst_2.11:jar:2.4.2:provided
[INFO] |  |  |  +- org.codehaus.janino:janino:jar:3.0.9:provided
[INFO] |  |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.9:provided
[INFO] |  |  |  \- org.antlr:antlr4-runtime:jar:4.7:provided
[INFO] |  |  +- org.apache.orc:orc-core:jar:nohive:1.5.5:provided
[INFO] |  |  |  +- org.apache.orc:orc-shims:jar:1.5.5:provided
[INFO] |  |  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:provided
[INFO] |  |  |  +- commons-lang:commons-lang:jar:2.6:provided
[INFO] |  |  |  \- io.airlift:aircompressor:jar:0.10:provided
[INFO] |  |  +- org.apache.orc:orc-mapreduce:jar:nohive:1.5.5:provided
[INFO] |  |  +- org.apache.parquet:parquet-column:jar:1.10.1:provided
[INFO] |  |  |  +- org.apache.parquet:parquet-common:jar:1.10.1:provided
[INFO] |  |  |  \- org.apache.parquet:parquet-encoding:jar:1.10.1:provided
[INFO] |  |  +- org.apache.parquet:parquet-hadoop:jar:1.10.1:provided
[INFO] |  |  |  +- org.apache.parquet:parquet-format:jar:2.4.0:provided
[INFO] |  |  |  \- org.apache.parquet:parquet-jackson:jar:1.10.1:provided
[INFO] |  |  \- org.apache.arrow:arrow-vector:jar:0.10.0:provided
[INFO] |  |     +- org.apache.arrow:arrow-format:jar:0.10.0:provided
[INFO] |  |     +- org.apache.arrow:arrow-memory:jar:0.10.0:provided
[INFO] |  |     +- com.carrotsearch:hppc:jar:0.7.2:provided
[INFO] |  |     \- com.vlkan:flatbuffers:jar:1.2.0-3f79e055:provided
[INFO] |  +- com.databricks:spark-xml_2.11:jar:0.5.0:compile
[INFO] |  +- org.json4s:json4s-ext_2.11:jar:3.5.3:compile
[INFO] |  |  +- joda-time:joda-time:jar:2.9.5:provided
[INFO] |  |  \- org.joda:joda-convert:jar:1.8.1:compile
[INFO] |  +- org.scalaz:scalaz-core_2.11:jar:7.2.29:compile
[INFO] |  \- org.scalaj:scalaj-http_2.11:jar:2.4.1:compile
[INFO] +- org.scala-lang:scala-library:jar:2.11.12:provided
[INFO] +- org.scalatest:scalatest_2.11:jar:3.1.0:test
[INFO] |  +- org.scalatest:scalatest-compatible:jar:3.1.0:test
[INFO] |  +- org.scalactic:scalactic_2.11:jar:3.1.0:test
[INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.12:provided
[INFO] |  \- org.scala-lang.modules:scala-xml_2.11:jar:1.2.0:provided
[INFO] +- org.scalatestplus:mockito-1-10_2.11:jar:3.1.0.0:test
[INFO] |  \- org.mockito:mockito-core:jar:3.2.0:test
[INFO] |     +- net.bytebuddy:byte-buddy:jar:1.10.3:test
[INFO] |     +- net.bytebuddy:byte-buddy-agent:jar:1.10.3:test
[INFO] |     \- org.objenesis:objenesis:jar:2.6:provided
[INFO] \- com.google.code.findbugs:jsr305:jar:3.0.2:provided
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.027 s
[INFO] Finished at: 2020-06-01T10:02:00+02:00
[INFO] ------------------------------------------------------------------------
uday1409 commented 4 years ago

If I use 0.5.3 packaged agent-core jar from maven repo, its working fine.

But, when I build core jar locally, its not working at all.

some weird observations:

when bundle is packaged, with agent-core 0.5.1 , size was 13.5 MB (worked ) when bundle is packaged, with agent-core 0.5.2 , size was 28.5 MB (did not work) when bundle is packaged, with agent-core 0.5.3 , size was 13.5 MB (worked ) and when bundle is packaged, with agent-core locally , size was 28.5 MB (did not work)

Seems like the Agent-core code/ parent pom that I took from spark-agent repo has code similar to agent-core-0.5.2 version in maven repo.

I wanted to understand the changes you made for 0.5.3 ..Could you help me with this one

wajda commented 4 years ago

There wasn't any significant change between those versions, only some bugfixes here and there and a few new API methods exposed. Really nothing that could affect the bundle size or dependencies. The bundle size for 0.5.2 is also around 13.5Mb as for the rest bundle versions. So I bet it's a packaging issue on your side. Try mvn clean install again to be sure.

uday1409 commented 4 years ago

Yes @wajda . I have done multiple times. Not sure where its going wrong. The size I mentioned is when I packaged locally with different versions of agent-core. Ran with mvn scala-cross-build:change-version -Pscala-2.11 too in project. No clue.

wajda commented 4 years ago

It looks like your bundle contains two agent-core's inside, probably compiled against different Scala version, which very likely is causing the issue. Please double check your build setup. Can you share the changes you've done to POMs? And exact commands you execute?

uday1409 commented 4 years ago

I will take a new copy of the code ,make those changes and will share with you.

uday1409 commented 4 years ago

@wajda , I have followed below steps to package locally. ( code under 5 and 6 points is not visible)

1) downloaded the code from (https://github.com/AbsaOSS/spline-spark-agent) develop branch

2) Opened it in Intellij IDE

3)Packaged agent core jar locally. Before that checked whether root Pom.xml has correct scala version . It was pointing to 2.11 only

4) Renamed core to have different artifact name and deployed locally using below command

 mvn deploy:deploy-file -DgroupId=za.co.absa.spline.agent.spark_1 -DartifactId=agent-core_2.11_1 -Dversion=0.5.2 -Durl=file:./local-maven-repo/ -DrepositoryId=local-maven-repo -DupdateReleaseInfo=true -Dfile=C:\Users\xxxx\Source\Repos\spline-spark-agent-develop\core\target\agent-core_2.11-0.6.0-SNAPSHOT.jar

5) In bundle-2.4 module pom file, added below repo

 <repositories>
    <repository>
        <id>local-maven-repo</id>
        <url>file:///C:/Users/udkallem/Source/Repos/spline-spark-agent-develop/local-maven-repo</url></repository> </repositories>

6) Commented maven repo reference to agent-core, and used locally deployed one. Please notice _1

<!--  <dependencies>
    <dependency>
        <groupId>za.co.absa.spline.agent.spark</groupId>
        <artifactId>agent-core_${scala.binary.version}</artifactId>
        <version>${project.version}</version>
    </dependency>
</dependencies>-->

<dependencies>
    <dependency>
        <groupId>za.co.absa.spline.agent.spark_1</groupId>
        <artifactId>agent-core_2.11_1</artifactId>
        <version>0.5.2</version>
    </dependency>
</dependencies>

7) Then packaged locally, the size shot up to 28MB , and when tested in databricks cluster, it throws same error.

8) I uncommented the reference to maven repo, and removed local dependency of agent core, the size was 13 MB and it worked.

Not sure what and where I did wrong. Could you please help me on this.

wajda commented 4 years ago

From what you described it should work. Did you compare the bundles content? Did you run mvn clean when building a bundle?

uday1409 commented 4 years ago

Yes, Alex @wajda . I did that multiple times. Need to clone and try again. Spark Agent Core code should by default supports 2.11 right

wajda commented 4 years ago

yes, it's 2.11 by default. Your case is very strange, Try to compare the contents of the correct and the wrong bundle jars to see what is that extra content in a bigger jar. Also run mvn dependency:tree and mvn help:effective-pom for those bundles and compare them as well. Hope it will shed some light.

uday1409 commented 4 years ago

I have run mvn dependency:tree with bundle pom pointing to the agent-core which is maven and deployed to local repo. I am not seeing lot of dependencies listed when I pointed local agent-core.

Bundle POM dependencies when pointed local agent-core

   maven-dependency-plugin:2.8:tree (default-cli) @ spark-2.4-spline-agent-bundle_2.11 ---
   [INFO] za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.11:jar:0.6.0-SNAPSHOT
   [INFO] +- za.co.absa.spline.agent.spark:agent-core_2.11:jar:0.5.6:compile
   [INFO] +- org.scala-lang:scala-library:jar:2.11.12:provided
   [INFO] +- org.scalatest:scalatest_2.11:jar:3.1.0:test
   [INFO] |  +- org.scalatest:scalatest-compatible:jar:3.1.0:test
   [INFO] |  +- org.scalactic:scalactic_2.11:jar:3.1.0:test
   [INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.12:provided
   [INFO] |  \- org.scala-lang.modules:scala-xml_2.11:jar:1.2.0:provided
   [INFO] +- org.scalatestplus:mockito-1-10_2.11:jar:3.1.0.0:test
   [INFO] |  \- org.mockito:mockito-core:jar:3.2.0:test

Bundle POM dependencies when pointed maven agent-core 0.5.6

  --- maven-dependency-plugin:2.8:tree (default-cli) @ spark-2.4-spline-agent-bundle_2.11 ---
  [INFO] za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.11:jar:0.6.0-SNAPSHOT
  [INFO] +- za.co.absa.spline.agent.spark:agent-core_2.11:jar:0.5.3:compile
  [INFO] |  +- org.scala-lang:scala-compiler:jar:2.11.12:provided
  [INFO] |  |  \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:provided
  [INFO] |  +- za.co.absa.commons:commons_2.11:jar:0.0.13:compile
  [INFO] |  +- org.apache.spark:spark-core_2.11:jar:2.4.2:provided
  [INFO] |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.8:provided
  [INFO] |  |  +- org.apache.avro:avro:jar:1.8.2:provided
  [INFO] |  |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided
  [INFO] |  |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided
  [INFO] |  |  |  +- org.apache.commons:commons-compress:jar:1.8.1:provided
  [INFO] |  |  |  \- org.tukaani:xz:jar:1.5:provided
  [INFO] |  |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.2:provided
  [INFO] |  |  |  +- org.apache.avro:avro-ipc:jar:1.8.2:provided
  [INFO] |  |  |  \- commons-codec:commons-codec:jar:1.9:provided
  [INFO] |  |  +- com.twitter:chill_2.11:jar:0.9.3:provided
  [INFO] |  |  |  \- com.esotericsoftware:kryo-shaded:jar:4.0.2:provided
  [INFO] |  |  |     \- com.esotericsoftware:minlog:jar:1.3.0:provided
  [INFO] |  |  +- com.twitter:chill-java:jar:0.9.3:provided
  [INFO] |  |  +- org.apache.xbean:xbean-asm6-shaded:jar:4.8:provided
  [INFO] |  |  +- org.apache.hadoop:hadoop-client:jar:2.6.5:provided
  [INFO] |  |  |  +- org.apache.hadoop:hadoop-common:jar:2.6.5:provided
  [INFO] |  |  |  |  +- commons-cli:commons-cli:jar:1.2:provided
  [INFO] |  |  |  |  +- xmlenc:xmlenc:jar:0.52:provided
  [INFO] |  |  |  |  +- commons-httpclient:commons-httpclient:jar:3.1:provided
  [INFO] |  |  |  |  +- commons-io:commons-io:jar:2.4:provided
  [INFO] |  |  |  |  +- commons-collections:commons-collections:jar:3.2.2:provided
  [INFO] |  |  |  |  +- commons-configuration:commons-configuration:jar:1.6:provided
  [INFO] |  |  |  |  |  +- commons-digester:commons-digester:jar:1.8:provided
  [INFO] |  |  |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:provided
  [INFO] |  |  |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:provided
  [INFO] |  |  |  |  +- com.google.code.gson:gson:jar:2.2.4:provided
  [INFO] |  |  |  |  +- org.apache.hadoop:hadoop-auth:jar:2.6.5:provided
  [INFO] |  |  |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.2.5:provided
  [INFO] |  |  |  |  |  |  \- org.apache.httpcomponents:httpcore:jar:4.2.4:provided
  [INFO] |  |  |  |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:provided
  [INFO] |  |  |  |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:provided
  [INFO] |  |  |  |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:provided
  [INFO] |  |  |  |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:provided
  [INFO] |  |  |  |  +- org.apache.curator:curator-client:jar:2.6.0:provided
  [INFO] |  |  |  |  \- org.htrace:htrace-core:jar:3.0.4:provided
  [INFO] |  |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.6.5:provided
  [INFO] |  |  |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:provided
  [INFO] |  |  |  |  \- xerces:xercesImpl:jar:2.9.1:provided
  [INFO] |  |  |  |     \- xml-apis:xml-apis:jar:1.3.04:provided
  [INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.5:provided
  [INFO] |  |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.5:provided
  [INFO] |  |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.5:provided
  [INFO] |  |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.5:provided
  [INFO] |  |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.5:provided
  [INFO] |  |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.5:provided
  [INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.5:provided
  [INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.5:provided
  [INFO] |  |  |  |     +- javax.xml.bind:jaxb-api:jar:2.2.2:provided
  [INFO] |  |  |  |     |  \- javax.xml.stream:stax-api:jar:1.0-2:provided
  [INFO] |  |  |  |     +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:provided
  [INFO] |  |  |  |     \- org.codehaus.jackson:jackson-xc:jar:1.9.13:provided
  [INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.5:provided
  [INFO] |  |  |  \- org.apache.hadoop:hadoop-annotations:jar:2.6.5:provided
  [INFO] |  |  +- org.apache.spark:spark-launcher_2.11:jar:2.4.2:provided
  [INFO] |  |  +- org.apache.spark:spark-kvstore_2.11:jar:2.4.2:provided
  [INFO] |  |  |  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
  [INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-core:jar:2.6.7:provided
  [INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.7:provided
  [INFO] |  |  +- org.apache.spark:spark-network-common_2.11:jar:2.4.2:provided
  [INFO] |  |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.4.2:provided
  [INFO] |  |  +- org.apache.spark:spark-unsafe_2.11:jar:2.4.2:provided
  [INFO] |  |  +- javax.activation:activation:jar:1.1.1:provided
  [INFO] |  |  +- org.apache.curator:curator-recipes:jar:2.6.0:provided
  [INFO] |  |  |  +- org.apache.curator:curator-framework:jar:2.6.0:provided
  [INFO] |  |  |  \- com.google.guava:guava:jar:16.0.1:provided
  [INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
  [INFO] |  |  +- javax.servlet:javax.servlet-api:jar:3.1.0:provided
  [INFO] |  |  +- org.apache.commons:commons-lang3:jar:3.5:provided
  [INFO] |  |  +- org.apache.commons:commons-math3:jar:3.4.1:provided
  [INFO] |  |  +- org.slf4j:slf4j-api:jar:1.7.16:provided
  [INFO] |  |  +- org.slf4j:jul-to-slf4j:jar:1.7.16:provided
  [INFO] |  |  +- org.slf4j:jcl-over-slf4j:jar:1.7.16:provided
  [INFO] |  |  +- log4j:log4j:jar:1.2.17:provided
  [INFO] |  |  +- org.slf4j:slf4j-log4j12:jar:1.7.16:provided
  [INFO] |  |  +- com.ning:compress-lzf:jar:1.0.3:provided
  [INFO] |  |  +- org.xerial.snappy:snappy-java:jar:1.1.7.3:provided
  [INFO] |  |  +- org.lz4:lz4-java:jar:1.4.0:provided
  [INFO] |  |  +- com.github.luben:zstd-jni:jar:1.3.2-2:provided
  [INFO] |  |  +- org.roaringbitmap:RoaringBitmap:jar:0.7.45:provided
  [INFO] |  |  |  \- org.roaringbitmap:shims:jar:0.7.45:provided
  [INFO] |  |  +- commons-net:commons-net:jar:3.1:provided
  [INFO] |  |  +- org.json4s:json4s-jackson_2.11:jar:3.5.3:provided
  [INFO] |  |  |  \- org.json4s:json4s-core_2.11:jar:3.5.3:provided
  [INFO] |  |  |     +- org.json4s:json4s-ast_2.11:jar:3.5.3:provided
  [INFO] |  |  |     \- org.json4s:json4s-scalap_2.11:jar:3.5.3:provided
  [INFO] |  |  +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:provided
  [INFO] |  |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:provided
  [INFO] |  |  |  +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:provided
  [INFO] |  |  |  |  +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:provided
  [INFO] |  |  |  |  \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:provided
  [INFO] |  |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:provided
  [INFO] |  |  |  \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:provided
  [INFO] |  |  |     \- org.javassist:javassist:jar:3.18.1-GA:provided
  [INFO] |  |  +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:provided
  [INFO] |  |  |  +- javax.annotation:javax.annotation-api:jar:1.2:provided
  [INFO] |  |  |  +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:provided
  [INFO] |  |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:provided
  [INFO] |  |  +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:provided
  [INFO] |  |  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:provided
  [INFO] |  |  |  \- javax.validation:validation-api:jar:1.1.0.Final:provided
  [INFO] |  |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:provided
  [INFO] |  |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:provided
  [INFO] |  |  +- io.netty:netty-all:jar:4.1.17.Final:provided
  [INFO] |  |  +- io.netty:netty:jar:3.9.9.Final:provided
  [INFO] |  |  +- com.clearspring.analytics:stream:jar:2.7.0:provided
  [INFO] |  |  +- io.dropwizard.metrics:metrics-core:jar:3.1.5:provided
  [INFO] |  |  +- io.dropwizard.metrics:metrics-jvm:jar:3.1.5:provided
  [INFO] |  |  +- io.dropwizard.metrics:metrics-json:jar:3.1.5:provided
  [INFO] |  |  +- io.dropwizard.metrics:metrics-graphite:jar:3.1.5:provided
  [INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.7.1:provided
  [INFO] |  |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.7.1:provided
  [INFO] |  |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.7.9:provided
  [INFO] |  |  +- org.apache.ivy:ivy:jar:2.4.0:provided
  [INFO] |  |  +- oro:oro:jar:2.0.8:provided
  [INFO] |  |  +- net.razorvine:pyrolite:jar:4.13:provided
  [INFO] |  |  +- net.sf.py4j:py4j:jar:0.10.7:provided
  [INFO] |  |  +- org.apache.spark:spark-tags_2.11:jar:2.4.2:provided
  [INFO] |  |  +- org.apache.commons:commons-crypto:jar:1.0.0:provided
  [INFO] |  |  \- org.spark-project.spark:unused:jar:1.0.0:provided
  [INFO] |  +- org.apache.spark:spark-sql_2.11:jar:2.4.2:provided
  [INFO] |  |  +- com.univocity:univocity-parsers:jar:2.7.3:provided
  [INFO] |  |  +- org.apache.spark:spark-sketch_2.11:jar:2.4.2:provided
  [INFO] |  |  +- org.apache.spark:spark-catalyst_2.11:jar:2.4.2:provided
  [INFO] |  |  |  +- org.codehaus.janino:janino:jar:3.0.9:provided
  [INFO] |  |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.9:provided
  [INFO] |  |  |  \- org.antlr:antlr4-runtime:jar:4.7:provided
  [INFO] |  |  +- org.apache.orc:orc-core:jar:nohive:1.5.5:provided
  [INFO] |  |  |  +- org.apache.orc:orc-shims:jar:1.5.5:provided
  [INFO] |  |  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:provided
  [INFO] |  |  |  +- commons-lang:commons-lang:jar:2.6:provided
  [INFO] |  |  |  \- io.airlift:aircompressor:jar:0.10:provided
  [INFO] |  |  +- org.apache.orc:orc-mapreduce:jar:nohive:1.5.5:provided
  [INFO] |  |  +- org.apache.parquet:parquet-column:jar:1.10.1:provided
  [INFO] |  |  |  +- org.apache.parquet:parquet-common:jar:1.10.1:provided
  [INFO] |  |  |  \- org.apache.parquet:parquet-encoding:jar:1.10.1:provided
  [INFO] |  |  +- org.apache.parquet:parquet-hadoop:jar:1.10.1:provided
  [INFO] |  |  |  +- org.apache.parquet:parquet-format:jar:2.4.0:provided
  [INFO] |  |  |  \- org.apache.parquet:parquet-jackson:jar:1.10.1:provided
  [INFO] |  |  \- org.apache.arrow:arrow-vector:jar:0.10.0:provided
  [INFO] |  |     +- org.apache.arrow:arrow-format:jar:0.10.0:provided
  [INFO] |  |     +- org.apache.arrow:arrow-memory:jar:0.10.0:provided
  [INFO] |  |     +- com.carrotsearch:hppc:jar:0.7.2:provided
  [INFO] |  |     \- com.vlkan:flatbuffers:jar:1.2.0-3f79e055:provided
  [INFO] |  +- com.databricks:spark-xml_2.11:jar:0.5.0:compile
  [INFO] |  +- org.json4s:json4s-ext_2.11:jar:3.5.3:compile
  [INFO] |  |  +- joda-time:joda-time:jar:2.9.5:provided
  [INFO] |  |  \- org.joda:joda-convert:jar:1.8.1:compile
  [INFO] |  +- org.scalaz:scalaz-core_2.11:jar:7.2.29:compile
  [INFO] |  \- org.scalaj:scalaj-http_2.11:jar:2.4.1:compile
  [INFO] +- org.scala-lang:scala-library:jar:2.11.12:provided
  [INFO] +- org.scalatest:scalatest_2.11:jar:3.1.0:test
  [INFO] |  +- org.scalatest:scalatest-compatible:jar:3.1.0:test
  [INFO] |  +- org.scalactic:scalactic_2.11:jar:3.1.0:test
  [INFO] |  +- org.scala-lang:scala-reflect:jar:2.11.12:provided
  [INFO] |  \- org.scala-lang.modules:scala-xml_2.11:jar:1.2.0:provided
  [INFO] +- org.scalatestplus:mockito-1-10_2.11:jar:3.1.0.0:test
  [INFO] |  \- org.mockito:mockito-core:jar:3.2.0:test
  [INFO] |     +- net.bytebuddy:byte-buddy:jar:1.10.3:test
  [INFO] |     +- net.bytebuddy:byte-buddy-agent:jar:1.10.3:test
  [INFO] |     \- org.objenesis:objenesis:jar:2.6:provided
  [INFO] \- com.google.code.findbugs:jsr305:jar:3.0.2:provided
  [INFO] ------------------------------------------------------------------------
  [INFO] BUILD SUCCESS
  [INFO] ------------------------------------------------------------------------
  [INFO] Total time:  5.109 s
  [INFO] Finished at: 2020-06-03T19:00:44+05:30
  [INFO] ------------------------------------------------------------------------
wajda commented 4 years ago

You are confusing versions. I see the the first POM depends on Spline agent-core 0.5.6 and the second one 0.5.3 version.

Try to use the same agent-core version in both cases, so you can compare the dependency trees and find the difference.

uday1409 commented 4 years ago

The version 0.5.6 was deployed to my local maven repo after packaging agent-core jar locally.

uday1409 commented 4 years ago

I have figured out the issue. Thank you for all your help @wajda . The issue was with databricks cluster itself. I will post my requirement in another issue.