search

drubbo / SparkGIS

GIS extension for SparkSQL
Apache License 2.0
37 stars 14 forks source link

[Spark 1.6.0]ClassCastException: java.lang.String cannot be cast to org.apache.spark.unsafe.types.UTF8String #8

Closed netanel246 closed 8 years ago

netanel246 commented 8 years ago

I read two geojson files(lines and points) as dataframes, then i created buffer dataframe from the points. I can see the geometries, but when i tried to do join between them and print the result i got this exception. The data is valid, because before the join i can print the features of the layers well. i tried to add option("charset", "UTF8") and also UTF16. I saw this error in other package: https://github.com/databricks/spark-csv/issues/206 https://github.com/databricks/spark-csv/issues/64

I am running it on windows, but connect to remote HDFS. In Addition, i compiled the package(SparkGIS) to scala 2.10.5 instead of using scala 2.11 (All the tests pass)

drubbo commented 8 years ago

Can you provide some excerpt of your code and data for a reproduction ?

netanel246 commented 8 years ago

Hi, Thank you for answering so quickly. I will add the it tomorrow. Thanks. On Mar 30, 2016 2:50 PM, "Emiliano Leporati" notifications@github.com wrote:

Can you provide some excerpt of your code and data for a reproduction ?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/drubbo/SparkGIS/issues/8#issuecomment-203394913

netanel246 commented 8 years ago

Update: I wasn't able to recreate the bug, so i close it.

jomach commented 7 years ago

I' having similar issues when creating a custom datasource.

Schema: StructType(Seq(StructField("Date", LongType), StructField("Device", StringType), StructField("Tag", StringType), StructField("TagValue", DoubleType)) )

Stack ob aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 2, localhost): scala.MatchError: 1451610000 (of class java.lang.Long) at org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:295) at org.apache.spark.sql.catalyst.CatalystTypeConverters$StringConverter$.toCatalystImpl(CatalystTypeConverters.scala:294) at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102) at org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$2.apply(CatalystTypeConverters.scala:401) at org.apache.spark.sql.execution.RDDConversions$$anonfun$rowToRowRdd$1$$anonfun$apply$2.apply(ExistingRDD.scala:59) at org.apache.spark.sql.execution.RDDConversions$$anonfun$rowToRowRdd$1$$anonfun$apply$2.apply(ExistingRDD.scala:56) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:389) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)

ronakhingar commented 5 years ago

I am also facing the same issue while creating a custom data source.

saikiriti93 commented 5 years ago

Is there some update on this?