awslabs / deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Apache License 2.0
3.32k stars 539 forks source link

Tammruka/2.0.0 spark 3.2.0 #403

Open joesan opened 2 years ago

joesan commented 2 years ago

Issue #, if available: https://github.com/awslabs/deequ/issues/380

Description of changes: Support for Scala 2.13 and Spark 3.2

It is not fully done as I face two issues before I could even successfully compile. Here is the result of the output:

joesan@joesan-S-14-v5:~/Projects/Private/scala-projects/deequ$ mvn clean install
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------------< com.amazon.deequ:deequ >-----------------------
[INFO] Building deequ 2.0.0-spark-3.2
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ deequ ---
[INFO] Deleting /home/joesan/Projects/Private/scala-projects/deequ/target
[INFO] 
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ deequ ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/joesan/Projects/Private/scala-projects/deequ/src/main/resources
[INFO] 
[INFO] --- scala-maven-plugin:4.5.6:add-source (scala-compile-first) @ deequ ---
[INFO] 
[INFO] --- scala-maven-plugin:4.5.6:compile (scala-compile-first) @ deequ ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /home/joesan/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.5.8-bin_2.13.0__52.0-1.5.8_20211211T222914.jar
[INFO] compiling 106 Scala sources and 1 Java source to /home/joesan/Projects/Private/scala-projects/deequ/target/classes ...
[ERROR] /home/joesan/Projects/Private/scala-projects/deequ/src/main/scala/com/amazon/deequ/analyzers/QuantileNonSample.scala:54: missing argument list for method apply in trait IterableFactory
Unapplied methods are only converted to functions when a function type is expected.
You can make this conversion explicit by writing `apply _` or `apply(_)` instead of `apply`.
[ERROR] one error found
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11.392 s
[INFO] Finished at: 2021-12-20T07:12:16+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.5.6:compile (scala-compile-first) on project deequ: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.5.6:compile failed: Compilation failed: InterfaceCompileFailed -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

vasilyevskiy commented 2 years ago

Any updates about this PR?