search

yc-huang / Hive-mongo

hive storage handler for connecting with MongoDB
Apache License 2.0
32 stars 33 forks source link

Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler #5

Open cooldude12 opened 12 years ago

cooldude12 commented 12 years ago

Hi, Thanks for the hive plug in. I ran into some issue while following readme to test the installation.
I am getting error in loading the storage handler. It may be a path or some environment variable issue.

I verified : /src/main/java/org/yong3/hive/mongo/MongoStorageHandler.java exists.

mongo_hive home dir: /home/ben/tools/mongo_hive

here is the full log

[DEV] ben@bi-all:~/tools$ $HIVE_HOME/bin/hive --auxpath /home//ben/tools/mongo_hive/mongo-java-driver-2.6.3.jar,/home/ben/tools/mongo_hive/guava-r06.jar, /home/ben/tools/mongo_hive/hive-mongo-0.0.1-SNAPSHOT.jar Logging initialized using configuration in jar:file:/home/ben/tools/hive-0.8.1-bin/lib/hive-common-0.8.1.jar!/hive-log4j.properties Hive history file=/tmp/ben/hive_job_log_ben_201207211824_1059679816.txt hive> create external table mongo_events(id int, dt int, ip string, event string) stored by "org.yong3.hive.mongo.MongoStorageHandler" with serdeproperties ( "mongo.column.mapping" = "_id,dt,ip,event" ) tblproperties ( "mongo.host" = "bi-fb-app.dev", "mongo.port" = "27017", "mongo.db" = "game", "mongo.collection" = "event_20120703" ); Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

yc-huang commented 12 years ago

could you help to post what's the error message in your hive log (which is located in /tmp//hive.log)?

thx, YC

cooldude12 commented 12 years ago

YC, Thanks for the reply. I think I missed the step of building all the jar files. Since I don't have a java back ground here are my questions:

  1. Do I need to build all thrre jar files? or can download it. I am running ubuntu mongo-java-driver-2.6.3.jar, guava-r06.jar, hive-mongo-0.0.1-SNAPSHOT.jar or I can download first two of them? Do you have a documentation on the all the build of each of this?

I see you have some answer for the snapshot: https://github.com/yc-huang/Hive-mongo/issues/3. Is the snapshot.jar is for files which you have created [i.e under mongo_hive/src/main/**...

for guava*jatr, I downloaded guava-r06-site.jar, will that work? or i need to do complete build after downloading all the source files

for mongo java driver, i downloaded the tar files. How do i make jar out of it. mongodb-mongo-java-driver-674c521

  1. After the building jar is completed, should i be able to do just use your readme doc? do in eed to set any other env variables?
  2. What are the limitation of existing package. Does it support group by etc.
  3. Any opinion on mongo-hadoop package. Do you know when 10gen is going to support hive. We are storing good amount of data in mongo, and all we care about is running hive. I suppose it will be faster than the regular hadoop+hive as mongo is faster on map-reduce than hadoop. Do you think so?

thanks a lot for your help

cooldude12 commented 12 years ago

since you asked for hive log, here is it

012-07-23 00:24:33,293 WARN conf.HiveConf (HiveConf.java:(68)) - hive-site.xml not found on CLASSPATH 2012-07-23 00:24:38,765 ERROR exec.Task (SessionState.java:printError(380)) - Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:278) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3360) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:225) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:151) at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:273) ... 17 more Caused by: java.lang.ClassNotFoundException: org.yong3.hive.mongo.MongoStorageHandler at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:144) ... 18 more

2012-07-23 00:24:38,765 ERROR ql.Driver (SessionState.java:printError(380)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask 2012-07-23 00:34:26,314 WARN conf.HiveConf (HiveConf.java:(68)) - hive-site.xml not found on CLASSPATH 2012-07-23 00:34:29,517 ERROR exec.Task (SessionState.java:printError(380)) - Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.yong3.hive.mongo.MongoStorageHandler at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:278) at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3360) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:225) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) "hive.log" 111L, 8379C

yc-huang commented 11 years ago

Sorry for miss your email. Now I include a combo jar under 'target' directory you might try without needing guava/mongo-java-driver jar...