search

harsha2010 / magellan

Geo Spatial Data Analytics on Spark
Apache License 2.0
534 stars 149 forks source link

Problem while converting coordinates to point in Spark 2.3.1 and Scala 2.11 #229

Closed djpirra closed 5 years ago

djpirra commented 5 years ago

Using this code in DF:

.withColumn("pickup_point", point($"pickup_longitude",$"pickup_latitude"))

I am having this exception: NoSuchMethodException: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext.addMutableState(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V at org.apache.spark.sql.types.PointConverter.doGenCode(PointConverter.scala:43) at org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:111) at org.apache.spark.sql.catalyst.expressions.Expression$$anonfun$genCode$2.apply(Expression.scala:108) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:108) at org.apache.spark.sql.catalyst.expressions.Alias.genCode(namedExpressions.scala:143) at org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext$$anonfun$24$$anonfun$apply$5.apply(CodeGenerator.scala:1367) at org.apache.spark.sql.catalyst.expressions.codegen.CodegenContext$$anonfun$24$$anonfun$apply$5.apply(CodeGenerator.scala:1366) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)

Is this a know bug? How to overcome?

djpirra commented 5 years ago

This is fixed. Needed to compile with SBT and use the latest version.

ziedbouf commented 5 years ago

Hello @djpirra i compiled everything from master branch but i still gets similar problem, could you share more details and if any specific configurations needed.

seanlindo commented 5 years ago

I have compiled from the master branch and tagged it as 1.06. Does not work any longer (Databricks). Given the repetitive comments, I think this is an actual issue.