search

cloudera-labs / envelope

Build configuration-driven ETL pipelines on Apache Spark
Apache License 2.0
157 stars 89 forks source link

Exclude jackson-databind from shaded jar #50

Open jrkinley opened 3 years ago

jrkinley commented 3 years ago

Exclude jackson-databind from shaded .jar as it clashes with Spark's version in Cloudera Runtime 7.1.x. Looking at the dependency tree Spark includes jackson-databind:2.6.5 whereas Envelope 0.7.2 is including jackson-databind:2.3.1 as part of a kitesdk dependency.

jrkinley commented 3 years ago

Spark throws this exception when both versions of the jar are on the classpath:

Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.readerFor(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/ObjectReader;
    at org.apache.hadoop.hive.common.StatsSetupConst$ColumnStatsAccurate.<clinit>(StatsSetupConst.java:176)
    at org.apache.hadoop.hive.common.StatsSetupConst.parseStatsAcc(StatsSetupConst.java:353)
    at org.apache.hadoop.hive.common.StatsSetupConst.setBasicStatsState(StatsSetupConst.java:241)
    at org.apache.hadoop.hive.common.StatsSetupConst.setStatsStateForCreateTable(StatsSetupConst.java:345)
    at org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2443)
    at org.apache.hadoop.hive.ql.metadata.Hive.lambda$loadDynamicPartitions$4(Hive.java:2909)
    ... 4 more