add the json-serde on hive emr

[TonyWuLihu]

Hi rcongiu Have you encountered any issue with jar on emr hive thrifthivemeta ?
I have tried to upload the jar to the classpath of hive and could create the table with the json-serde class. But when i try using Thrift client to read metadata, the hive table could not read ,and the exception like below: Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.openx.data.jsonserde.JsonSerDe not found) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftH iveMetastore.java:35482) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftH iveMetastore.java:35449) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:35375) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:1017) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:1003) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1497) at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:80) at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:86) ... 84 more

[rcongiu]

The SerDe has to be avaible to the hive thrift service in its extra libs directory or in the classpath.Looks like it's not. R. ------------------------------------------------------- "Good judgment comes from experience. Experience comes from bad judgment"

---

On Monday, March 20, 2017 2:03 AM, Lihu Wu <notifications@github.com> wrote:

Hi rcongiu Have you encountered any issue with jar on emr hive thrifthivemeta ? I have tried to upload the jar to the classpath of hive and could create the table with the json-serde class. But when i try using Thrift client to read metadata, the hive table could not read ,and the exception like below: Caused by: MetaException(message:java.lang.ClassNotFoundException Class org.openx.data.jsonserde.JsonSerDe not found) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftH iveMetastore.java:35482) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftH iveMetastore.java:35449) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:35375) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:1017) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:1003) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1497) at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:80) at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:86) ... 84 more— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[TonyWuLihu]

Thanks for your reply! However, I checked the hive dependency and hive -e set, I could see the jar archive list in the classpath. Do I need to copy this jar to every node in my cluster in order to make it available to thrift service ?

[rcongiu]

You shouldn't need it, but there was a bug in the hive thrift service that would make it miss additional jars. You should ask in the hive-users mailing list, since it's an hive issue.See https://issues.apache.org/jira/browse/HIVE-5275 ------------------------------------------------------- "Good judgment comes from experience. Experience comes from bad judgment"

---

On Monday, March 20, 2017 8:28 AM, Lihu Wu <notifications@github.com> wrote:

Thanks for your reply! However, I checked the hive dependency and hive -e set, I could see the jar archive list in the classpath. Do I need to copy this jar to every node in my cluster in order to make it available to thrift service ?— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[TonyWuLihu]

Thanks for the info !!!