Esri / spatial-framework-for-hadoop

The Spatial Framework for Hadoop allows developers and data scientists to use the Hadoop data processing system for spatial data analysis.
Apache License 2.0
363 stars 160 forks source link

JSON SerDe on SparkSQL throws ClassCastException #97

Closed randallwhitman closed 7 years ago

randallwhitman commented 8 years ago

java.lang.ClassCastException: java.lang.Long cannot be cast to org.apache.hadoop.io.Writable

Seen in SparkSQL shell; had not been seen when testing on Hive. Both with v1.1 and master.

randallwhitman commented 8 years ago

For integration-regression testing, I ran the following queries updated from #25:

create external table trips_by_origin_cell(leftlon double, botlat double, rightlon double, toplat double, totnum int, samedest int, pct double, destlhs double, destbot double, destrhs double, desttop double)
row format delimited fields terminated by '\t'
location '/user/rwhitman/out-trips-by-origin-cell';

create external table trip_origin_json  (totnum int, samedest int, pct double, destlhs double, destbot double, destrhs double, desttop double, shape binary)
row format serde 'com.esri.hadoop.hive.serde.EsriJsonSerDe'
stored as inputformat 'com.esri.json.hadoop.UnenclosedEsriJsonInputFormat'
outputformat 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
;
insert overwrite table trip_origin_json select totnum, samedest, pct, destlhs, destbot, destrhs, desttop,
  ST_Polygon(leftlon,botlat, rightlon,botlat, rightlon,toplat, leftlon,toplat) from trips_by_origin_cell;

create external table trip_origin_geojs  (totnum int, samedest int, pct double, destlhs double, destbot double, destrhs double, desttop double, shape binary)
row format serde 'com.esri.hadoop.hive.serde.GeoJsonSerDe'
stored as inputformat 'com.esri.json.hadoop.UnenclosedGeoJsonInputFormat'
outputformat 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
;
insert overwrite table trip_origin_geojs select totnum, samedest, pct, destlhs, destbot, destrhs, desttop,
  ST_Polygon(leftlon,botlat, rightlon,botlat, rightlon,toplat, leftlon,toplat) from trips_by_origin_cell;