search

nexr / RHive

RHive is an R extension facilitating distributed computing via Apache Hive.
http://nexr.github.io/RHive
122 stars 63 forks source link

RUDF rhive.query failed due to serialisation exception #58

Open pkernevez opened 10 years ago

pkernevez commented 10 years ago

Hi,

We are trying to execute the exemples (https://github.com/nexr/RHive/wiki/RHive-example-code). When trying to execute the query, our jobs failed with a KryoException. It seems that an UDF instance is serialized even it contained converters that are not designed for serialization (no default constructor).

We are using Hadoop 2.2 and hive 0.12 (Horton distribution).

Are those examples still correct ? Do you have an idea of the cause of our error ?

Regards, Philippe

The exemple : coefficient <- 1.1 scoring <- function(sal) { coefficient * sal } rhive.assign('coefficient',coefficient) rhive.assign('scoring',scoring) rhive.exportAll(‘scoring’) rhive.query("select R('scoring',col_sal,0.0) from emp")

Exception : Error: java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter Serialization trace: converters (com.nexr.rhive.hive.udf.RUDF) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:314) at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:263) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:376) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:552) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:167) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:408) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter Serialization trace: converters (com.nexr.rhive.hive.udf.RUDF) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1097) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1109) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:526) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:502) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:367) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:276) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:810) at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:720) at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:733) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:287) ... 13 more

ssshow16 commented 10 years ago

Hi, Kernevez

PrimitiveObjectInspectorConverter$TextConverter class in Hive 0.12.0 don't have no-arg constructor. In other words, this problem is the bug in Hive 0.12.0

Similar situations happened in Hive 0.12.

  1. HIVE-5279
  2. HIVE-5779

I guess that these bugs will be fixed and released in hive-0.13.0.

Thanks! If there is any problem again, feel free to contact us.

On Tue, Mar 4, 2014 at 9:01 PM, Kernevez notifications@github.com wrote:

Hi,

We are trying to execute the exemples ( https://github.com/nexr/RHive/wiki/RHive-example-code). When trying to execute the query, our jobs failed with a KryoException. It seems that an UDF instance is serialized even it contained converters that are not designed for serialization (no default constructor).

We are using Hadoop 2.2 and hive 0.12 (Horton distribution).

Are those examples still correct ? Do you have an idea of the cause of our error ?

Regards, Philippe

The exemple :

coefficient <- 1.1 scoring <- function(sal) { coefficient * sal } rhive.assign('coefficient',coefficient) rhive.assign('scoring',scoring) rhive.exportAll('scoring') rhive.query("select R('scoring',col_sal,0.0) from emp")

Exception :

Error: java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter Serialization trace: converters (com.nexr.rhive.hive.udf.RUDF) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:314) at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:263) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:376) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:552) at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.(MapTask.java:167) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:408) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157) Caused by: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter Serialization trace: converters (com.nexr.rhive.hive.udf.RUDF) genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1097) at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1109) at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:526) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:502) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:367) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:276) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:810) at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:720) at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:733) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:287) ... 13 more

Reply to this email directly or view it on GitHubhttps://github.com/nexr/RHive/issues/58 .