search

awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Apache License 2.0
3.32k stars 539 forks source link

Scala 2.11.12 sbt deequ build fail #407

Closed pavannpa closed 2 years ago

pavannpa commented 2 years ago

Hello,

We have scala 2.11.12 and spark 2.4 setup as part of scala project. When deequ is included as part of build and sbt compile | sbt assembly is ran we are experiencing conflict versions . Tried excluding packages by using ExclusionRule or exclude . None of them worked

scala - 2.11.12 spark 2.4 deequ version - 1.2.2-spark-2.4

Build content: 

  lazy val deequDependencies = sparkDependencies("2.4.0") ++ Seq(
    "com.github.pureconfig"    %% "pureconfig"  % "0.7.2",
    "com.github.nikita-volkov"  % "sext"        % "0.2.4",
    "com.amazon.deequ"          % "deequ"       % "1.2.2-spark-2.4" % "provided"
      exclude("org.scalanlp", "breeze_2.11")
      exclude("com.chuusai", "shapeless_2.11")
      exclude("org.apache.spark", "spark-core_2.11")
      exclude("org.apache.spark", "spark-sql_2.11")

Error

error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/Users/pavannagaraja/Documents/repos/stash-data-source-ingestion/scala/"), "DeequValidator"): [error] org.apache.spark:spark-launcher _2.11, _2.12 [error] com.chuusai:shapeless _2.11, _2.12 [error] org.apache.spark:spark-sketch _2.11, _2.12 [error] org.apache.spark:spark-kvstore _2.11, _2.12 [error] org.json4s:json4s-ast _2.11, _2.12 [error] org.apache.spark:spark-catalyst _2.11, _2.12 [error] org.apache.spark:spark-network-shuffle _2.11, _2.12 [error] com.twitter:chill _2.11, _2.12 [error] org.apache.spark:spark-sql _2.11, _2.12 [error] org.scala-lang.modules:scala-xml _2.11, _2.12 [error] org.json4s:json4s-jackson _2.11, _2.12 [error] org.typelevel:macro-compat _2.11, _2.12 [error] com.fasterxml.jackson.module:jackson-module-scala _2.11, _2.12 [error] org.json4s:json4s-core _2.11, _2.12 [error] org.apache.spark:spark-unsafe _2.11, _2.12 [error] org.json4s:json4s-scalap _2.11, _2.12 [error] org.scala-lang.modules:scala-parser-combinators _2.11, _2.12 [error] org.apache.spark:spark-tags _2.11, _2.12 [error] org.apache.spark:spark-core _2.11, _2.12 [error] org.apache.spark:spark-network-common _2.11, _2.12 [error] java.lang.RuntimeException: Conflicting cross-version suffixes in: org.apache.spark:spark-launcher, com.chuusai:shapeless, org.apache.spark:spark-sketch, org.apache.spark:spark-kvstore, org.json4s:json4s-ast, org.apache.spark:spark-catalyst, org.apache.spark:spark-network-shuffle, com.twitter:chill, org.apache.spark:spark-sql, org.scala-lang.modules:scala-xml, org.json4s:json4s-jackson, org.typelevel:macro-compat, com.fasterxml.jackson.module:jackson-module-scala, org.json4s:json4s-core, org.apache.spark:spark-unsafe, org.json4s:json4s-scalap, org.scala-lang.modules:scala-parser-combinators, org.apache.spark:spark-tags, org.apache.spark:spark-core, org.apache.spark:spark-network-common