harsha2010 / magellan

Geo Spatial Data Analytics on Spark
Apache License 2.0
534 stars 149 forks source link

Tutorial Issue finding 'point' #241

Closed apiszcz closed 4 years ago

apiszcz commented 4 years ago

Following this tutorial https://magellan.ghost.io/magellan-geospatial-processing-made-easy/

I see this error

**``` command-3454590236003018:9: error: not found: value point point($"pickup_longitude",$"pickup_latitude"))



%scala
import magellan.Point
val trips = sqlContext.read
    .format("com.databricks.spark.csv")
    .option("comment", "V")
    .option("mode", "DROPMALFORMED")
    .schema(schema)
    .load("/FileStore/tables/nyc/trips/*")
    .withColumn("point", 
     point($"pickup_longitude",$"pickup_latitude"))
    .cache()
apiszcz commented 4 years ago

This example works correctly

%scala
import magellan.Point
import org.apache.spark.sql.functions.udf
val toPointUDF = udf{(x:Double,y:Double) => Point(x,y) }
import magellan.Point
import org.apache.spark.sql.functions.udf

%scala
val points = sc.parallelize(Seq((-1.0, -1.0), (-1.0, 1.0), (1.0, -1.0))).toDF("x", "y")
display(points.withColumn("point", toPointUDF('x, 'y)))
apiszcz commented 4 years ago

This resolved the issue

import magellan. import org.apache.spark.sql.magellan.dsl.expressions.

import org.apache.spark.sql.Row import org.apache.spark.sql.functions. import org.apache.spark.sql.types.