search

FINRAOS / MegaSparkDiff

A Spark-based data comparison tool at scale which facilitates software development engineers to compare a plethora of pair combinations of possible data sources. Multiple execution modes in multiple environments enable the user to generate a diff report as a Java/Scala-friendly DataFrame or as a file for future use. Comes with out of the box SparkFactory and SparkCompare tools.
https://finraos.github.io/MegaSparkDiff/
Apache License 2.0
49 stars 26 forks source link

Getting Errors #87

Open holson1980 opened 1 year ago

holson1980 commented 1 year ago

Hi, I have cloned this project and wanted to learn its capabilities. So, I ran a test case inside the examples folder and got error:

java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x7f63425a) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x7f63425a

Besides that, I was going to try comparing 2 CSV flies in locally and see differences in the outcome report. Could someone, please assist me with the issue? Thank you in advance!

matthewgillett commented 1 year ago

Hello, what version of Java are you running with?

Please see the following if you are using Java 17, which should help you get it running without error: https://stackoverflow.com/questions/73465937/apache-spark-3-3-0-breaks-on-java-17-with-cannot-access-class-sun-nio-ch-direct

holson1980 commented 1 year ago

Hi Matthew, I am using java version "18.0.1.1"