Support for Spark 2.4.3

[mscnop]

Hi, this is my first ever PR to anything, so please bear with me if I have done something terribly illegal/bad practice :)

I have updated a range of the dependencies to allow for Magellan compatibility with Spark 2.4.3 and Scala 2.11.12. In order to make it work with the new versions of the dependencies I had to change a few things regarding class/object creation.

All tests are passing, when I run "sbt test".

Best Mads

[codecov-io]

Codecov Report

Merging #238 into master will decrease coverage by 0.16%. The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master     #238      +/-   ##
==========================================
- Coverage   90.12%   89.96%   -0.17%     
==========================================
  Files          52       52              
  Lines        1782     1783       +1     
  Branches      125      124       -1     
==========================================
- Hits         1606     1604       -2     
- Misses        176      179       +3

Impacted Files	Coverage Δ
...in/scala/magellan/mapreduce/ShapeInputFormat.scala	90.62% <50%> (ø)	:arrow_up:
src/main/scala/magellan/catalyst/SpatialJoin.scala	93.02% <75%> (-2.22%)	:arrow_down:
src/main/scala/magellan/DoubleArrayData.scala	16.66% <0%> (-4.17%)	:arrow_down:
src/main/scala/magellan/IntegerArrayData.scala	16.66% <0%> (-4.17%)	:arrow_down:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 402df2f...de1ca19. Read the comment docs.

[khajaasmath786]

I am also looking for spark 2.4 support. getting below exceptions with 1.5 version of magellan.

Exception in thread "main" java.lang.VerifyError: class magellan.catalyst.Spatia lJoinHint overrides final method canonicalized.()Lorg/apache/spark/sql/catalyst/ plans/QueryPlan; at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:14 2) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

[khajaasmath786]

I was able to get this working with https://github.com/mscno/magellan for spark2.4 .

[harsha2010]

Does this preserve backward compatibility with Spark 2.3?

[khajaasmath786]

Does this preserve backward compatibility with Spark 2.3?

I just tested it out for spark2.1 and it doesn't work for it.

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.cat alyst.plans.logical.LogicalPlan.transformUp(Lscala/PartialFunction;)Lorg/apache/ spark/sql/catalyst/plans/logical/LogicalPlan; at magellan.catalyst.SpatialJoin.apply(SpatialJoin.scala:36) at magellan.catalyst.SpatialJoin.apply(SpatialJoin.scala:27) at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$ anonfun$apply$1.apply(RuleExecutor.scala:85) at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$ anonfun$apply$1.apply(RuleExecutor.scala:82) at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized .scala:124) at scala.collection.immutable.List.foldLeft(List.scala:84) at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.a pply(RuleExecutor.scala:82)

[mscnop]

Hi,

When i did the changes, I was able to make it work for both Spark 2.3.x and Spark 2.4.x. And both versions compile and test without errors in the CI travis builds: CI Results. I was not able to maintain backwards compatability with SPark 2.1.x or Spark 2.2.x. /Mads