search

harsha2010 / magellan

Geo Spatial Data Analytics on Spark
Apache License 2.0
534 stars 149 forks source link

What does this work with? #232

Closed seanlindo closed 5 years ago

seanlindo commented 5 years ago

At some point, I was able to manipulate enough levers to get this working with Spark 2.3.1 on Databricks. I'm not sure what's changed over the past month, but I'm getting the common NoSuchMethodError again.

Between the different versions of Spark, Scala, and Magellan 1.05 vs. 1.06, what combination actually works so we can get it documented?

harsha2010 commented 5 years ago

Ok. Databricks Spark is actually different from Apache Spark. Apache Spark 2.3.1 has been tested but not all the Databricks Runtime versions. It's going to be hard to support all DBR versions as they are not binary compatible with Apache Spark. The best we can do is support Apache Spark releases at some cadence and try and support the latest DBR LTS version. Would that work?

seanlindo commented 5 years ago

Alright, I'll try setting up and running my own Spark cluster. Thanks.

harsha2010 commented 5 years ago

Try using Apache Spark 2.3.1 on EMR. If you run into issues there, then we can look into it. Also 1.0.5 works with Spark 2.1 and 2.2 so if you are using 2.3, then use the master branch

seanlindo commented 5 years ago

Sounds good. I'll report back.

seanlindo commented 5 years ago

I can confirm this with with the following Docker Container running Spark 2.2.0. The issue must lie somewhere with the Databricks Runtime. Thanks.