search

harsha2010 / magellan

Geo Spatial Data Analytics on Spark
Apache License 2.0
534 stars 149 forks source link

Support Spark 2.3 #213

Closed NJordan72 closed 6 years ago

NJordan72 commented 6 years ago

Currently Spark doesn't work with Spark 2.3.

milanvdm commented 6 years ago

Same issue here. We are getting java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext.addMutableState(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;) when using 

val checkIsInCountry = udf { (point: Point, countryPolygons: Seq[Polygon]) =>
        countryPolygons.exists(p => {
          // Consider points that are explicitly contained in the shapes or to have some tolerance also consider those that are in the bounding boxes
          p.contains(point) || (
            point.getX() >= p.boundingBox.getXmin() && point.getX() <= p.boundingBox.getXmax() &&
            point.getY() >= p.boundingBox.getYmin() && point.getY() <= p.boundingBox.getYmax())
        })
      }

It works in Spark 2.2.1.

NJordan72 commented 6 years ago

@harsha2010 You mentioned in Gitter that you had a PR that would enable Spark 2.3 support. Is that something you can push? I've done some work to try to get it to work, but there is at least one test failing based on my branch. Happy to push that here as well if that is easier to work off of.

NJordan72 commented 6 years ago

Closed