s22s / pre-lt-raster-frames

Spark DataFrames for earth observation data
http://rasterframes.io
Other
19 stars 5 forks source link

EMR zeppelin `%sql` error with TIleUDT #50

Open vpipkt opened 6 years ago

vpipkt commented 6 years ago

Do this on EMR zep

val layer[astraea.spark.rasterframes.datasource.geotrellis.Layer] = ???
val rf = spark.read.geotrellis
    .loadRF(layer)
rf.createOrReplaceTempView("someTableFoo")

Then in next paragraph:

%sql
SELECT * FROM someTableFoo

You get:

org.apache.spark.SparkException: Can not load in UserDefinedType org.apache.spark.sql.gt.types.TileUDT for user class geotrellis.raster.Tile.
    at org.apache.spark.sql.types.UDTRegistration$$anonfun$getUDTFor$1.apply(UDTRegistration.scala:84)
    at org.apache.spark.sql.types.UDTRegistration$$anonfun$getUDTFor$1.apply(UDTRegistration.scala:73)
    at scala.Option.map(Option.scala:146)
    at org.apache.spark.sql.types.UDTRegistration$.getUDTFor(UDTRegistration.scala:73)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$.org$apache$spark$sql$catalyst$encoders$RowEncoder$$serializerFor(RowEncoder.scala:83)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$$anonfun$2.apply(RowEncoder.scala:160)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$$anonfun$2.apply(RowEncoder.scala:159)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:186)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$.org$apache$spark$sql$catalyst$encoders$RowEncoder$$serializerFor(RowEncoder.scala:159)
    at org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(RowEncoder.scala:61)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:68)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:632)
    at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:691)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.zeppelin.spark.SparkSqlInterpreter.interpret(SparkSqlInterpreter.java:116)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:97)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:498)
    at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
    at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
vpipkt commented 6 years ago

this is with raster-frames on commit bb060fbb56ef6686b7a793114be7a025311790e3

metasim commented 6 years ago

@vpipkt Any clue if this is still an issue? Not quite sure how this use case is any different from other standard use cases.

vpipkt commented 6 years ago

Will check this today with 0.6.0-RC7