search

zrlio / crail-spark-io

Fast I/O plugins for Spark
Apache License 2.0
41 stars 14 forks source link

Adapt code and dependencies to Spark 3.0.1 #12

Open asqasq opened 3 years ago

asqasq commented 3 years ago

Adapt the plugin tp Spark 3.0. The version for Spark 2.2.0 is under a new branch spark_2_2_0 so that we can keep the newest version for the newrest Spark version in master.

I have tested the plugin with Spark 3.0.1, Hadoop 2.7, Apache Crail 1.3 and Crail Spark Terasort with 1GB, 4GB, 16HB and 64GB and validated the correct sorting with and without this plugin. I did not run into problems or incorrect sortings.

Please have a look at the code.

PepperJo commented 3 years ago

Feel free to add yourself as an authors in https://github.com/zrlio/crail-spark-io/blob/master/AUTHORS