search

rcongiu / Hive-JSON-Serde

Read - Write JSON SerDe for Apache Hive.
Other
733 stars 391 forks source link

hive3.1.2 on hadoop 3.2.1 JsonSerDe #225

Open mrhjx2 opened 4 years ago

mrhjx2 commented 4 years ago

load json type data use JsonSerDe(1.3.8)

CREATE EXTERNAL TABLE test001( e string COMMENT 'from deserializer', p string COMMENT 'from deserializer', t timestamp COMMENT 'from deserializer', app string COMMENT 'from deserializer') PARTITIONED BY ( dt string) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' WITH SERDEPROPERTIES ( 'ignore.malformed.json'='true') STORED AS TEXTFILE; there'not any problem when load data to hive; but I get a issues exception msg on running SQL(select t from test001). the exception msg is : Exception in thread "main" java.lang.AbstractMethodError: Method org/openx/data/jsonserde/objectinspector/primitive/JavaStringTimestampObjectInspector.getPrimitiveWritableObject(Ljava/lang/Object;)Lorg/apache/hadoop/hive/serde2/io/TimestampWritableV2; is abstract at org.openx.data.jsonserde.objectinspector.primitive.JavaStringTimestampObjectInspector.getPrimitiveWritableObject(JavaStringTimestampObjectInspector.java) at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:308) at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:292) at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:247)

there is no problem when run SQL(select p from test001).

rcongiu commented 4 years ago

can you attach files with the table definition and the sample data that causes the error ? The create table is garbled and without the data you're running I can't reproduce the error.

rentaow commented 4 years ago

I can hit this issue as well. This is likely an incompatibility issue with Hive 3.x due to some model change in the hive timestamp object.

I can repro with the follow statement and running on Hive version 3.1.2

CREATE TABLE t1(
a string,
t timestamp)
ROW FORMAT SERDE
'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (
'ignore.malformed.json'='true');

insert into t1 select 'aaa',current_timestamp();

Status: Failed Vertex failed, vertexName=Map 1, vertexId=vertex_1598478619392_0002_1_00, diagnostics=[Task failed, taskId=task_1598478619392_0002_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : java.lang.RuntimeException: java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.objectinspector.primitive.TimestampObjectInspector.getPrimitiveJavaObject(Ljava/lang/Object;)Ljava/sql/Timestamp; at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)

rcongiu commented 4 years ago

thanks, I'll have a look