I have an Avro-backed table using a partition (the table create statement looks almost exactly like the example for the "kst" table with a partition in the Haivvreo readme).
Using Hive 0.7.1-cdh3u3 and Haivvreo master, when inserting dynamically into a partition, I get the following exception:
2012-04-26 16:00:10,801 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.RuntimeException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.SubStructObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:268) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:469) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.SubStructObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector at com.linkedin.haivvreo.AvroSerializer.serialize(AvroSerializer.java:44) at com.linkedin.haivvreo.AvroSerDe.serialize(AvroSerDe.java:129) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:544) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:87) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:730) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:733) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:841) at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:263) at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:198) ... 7 more
It started working after I changed the first line of AvroSerializer.serialize from this:
I have an Avro-backed table using a partition (the table create statement looks almost exactly like the example for the "kst" table with a partition in the Haivvreo readme).
Using Hive 0.7.1-cdh3u3 and Haivvreo master, when inserting dynamically into a partition, I get the following exception:
2012-04-26 16:00:10,801 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.RuntimeException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.SubStructObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:268) at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:469) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.SubStructObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector at com.linkedin.haivvreo.AvroSerializer.serialize(AvroSerializer.java:44) at com.linkedin.haivvreo.AvroSerDe.serialize(AvroSerDe.java:129) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:544) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:87) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:730) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:733) at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:841) at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:263) at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:198) ... 7 more
It started working after I changed the first line of AvroSerializer.serialize from this:StandardStructObjectInspector ssoi = (StandardStructObjectInspector) objectInspector;
to this:
StructObjectInspector ssoi = (StructObjectInspector) objectInspector;