search

jghoman / haivvreo

Hive + Avro. Serde for working with Avro in Hive
Apache License 2.0
59 stars 27 forks source link

Inserting into dynamic partition fails #17

Closed dkarvounis closed 12 years ago

dkarvounis commented 12 years ago

I have an Avro-backed table using a partition (the table create statement looks almost exactly like the example for the "kst" table with a partition in the Haivvreo readme).

Using Hive 0.7.1-cdh3u3 and Haivvreo master, when inserting dynamically into a partition, I get the following exception:

2012-04-26 16:00:10,801 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.RuntimeException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.SubStructObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:268) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:469) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.SubStructObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector at com.linkedin.haivvreo.AvroSerializer.serialize(AvroSerializer.java:44) at com.linkedin.haivvreo.AvroSerDe.serialize(AvroSerDe.java:129) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:544) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:87) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:730) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:733) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:841) at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:263) at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:198) ... 7 more It started working after I changed the first line of AvroSerializer.serialize from this:

StandardStructObjectInspector ssoi = (StandardStructObjectInspector) objectInspector;

to this:

StructObjectInspector ssoi = (StructObjectInspector) objectInspector;

jghoman commented 12 years ago

I had hit this a few days ago but hadn't had a time to track down the problem. Thanks a lot. Want to come up with a merge request?

jghoman commented 12 years ago

Merged in. Closing.