search

hbutani / SQLWindowing

SQL Windowing Functions for Hadoop
65 stars 17 forks source link

role of metaserver service #3

Closed achaubal closed 12 years ago

achaubal commented 12 years ago

Hi Harish,

looking at the windowing.pdf doc, it seemed that the metastore --service was needed to make this work for the windowing jar. Is it mandatory? or is it just a representative picture?

i.e. can I not have just a regular mysql running somewhere and make appropriate changes to hive-site.xml to point to that machine with mysql with appropriate jdbc driver? Or the metastore as a service is NEEDED by the windowing jar to make connections into A metastore.

thanks

ameet

hbutani commented 12 years ago

I have not tested this. But as far as I recall am not doing anything special to connect to the metastoreserver; just using the Hive client API. So my hope is it should just work if you point to mysql in hive-config. If you have a mysql based env, could you give it a quick test...

On 01/31/2012 09:25 AM, achaubal wrote:

Hi Harish,

looking at the windowing.pdf doc, it seemed that the metastore --service was needed to make this work for the windowing jar. Is it mandatory? or is it just a representative picture?

i.e. can I not have just a regular mysql running somewhere and make appropriate changes to hive-site.xml to point to that machine with mysql with appropriate jdbc driver? Or the metastore as a service is NEEDED by the windowing jar to make connections into A metastore.

thanks

ameet

Reply to this email directly or view it on GitHub: https://github.com/hbutani/SQLWindowing/issues/3

achaubal commented 12 years ago

actually, it's not working without metaserver service, I tried it

hive --service windowingCli -w /tmp/windowing/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar Hive history file=/tmp/root/hive_job_log_root_201202011345_1187471675.txt hive> show databases; FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask hive>

looks like by adding the --service and -w , the classpath is getting messed up may be??

hbutani commented 12 years ago

can you send me the complete stacktrace. the -w option should not interfere with regular hive operation (wmode hive) Can you confirm just hive is working... double check that it is not picking up the hive conf setting that point to a metastoreserver

achaubal commented 12 years ago

ok, I downloaded the new jar and tried to run against a mySQL metastore without --service clause. Still get the error. I have confirmed that without the windowing jar I am able to connect. Here is the stacktrace: hive> show databases; 12/02/02 15:17:57 DEBUG parse.VariableSubstitution: Substitution is on: show databases 12/02/02 15:17:57 INFO parse.ParseDriver: Parsing command: show databases 12/02/02 15:17:58 INFO parse.ParseDriver: Parse Completed 12/02/02 15:17:58 DEBUG fs.FileSystem: Creating filesystem for file:/// 12/02/02 15:17:58 INFO ql.Driver: Semantic Analysis Completed 12/02/02 15:17:58 DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.DelimitedJSONSerDe initialized with: columnNames=[] columnTypes=[] separator=[[B@443acc67] nullstring= lastColumnTakesRest=false 12/02/02 15:17:58 DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with: columnNames=[database_name] columnTypes=[string] separator=[[B@50fba502] nullstring= lastColumnTakesRest=false 12/02/02 15:17:58 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null) 12/02/02 15:17:58 DEBUG lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.DelimitedJSONSerDe initialized with: columnNames=[] columnTypes=[] separator=[[B@1e9cd8db] nullstring= lastColumnTakesRest=false 12/02/02 15:17:58 INFO ql.Driver: Starting command: show databases 12/02/02 15:17:58 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.cache.level2.type value null from jpox.properties with SOFT 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.plugin.pluginRegistryBundleCheck value null from jpox.properties with LOG 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding javax.jdo.option.ConnectionURL value null from jpox.properties with jdbc:mysql://bofa3/metastore 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.autoStartMechanismMode value null from jpox.properties with checked 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.validateConstraints value null from jpox.properties with false 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.autoCreateSchema value null from jpox.properties with false 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.cache.level2 value null from jpox.properties with false 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding javax.jdo.option.ConnectionUserName value null from jpox.properties with hiveuser 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding javax.jdo.option.DetachAllOnCommit value null from jpox.properties with true 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.connectionPoolingType value null from jpox.properties with DBCP 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.fixedDatastore value null from jpox.properties with true 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.storeManagerType value null from jpox.properties with rdbms 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding javax.jdo.option.ConnectionDriverName value null from jpox.properties with com.mysql.jdbc.Driver 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.transactionIsolation value null from jpox.properties with read-committed 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.validateColumns value null from jpox.properties with false 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding javax.jdo.option.NonTransactionalRead value null from jpox.properties with true 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.validateTables value null from jpox.properties with false 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding javax.jdo.PersistenceManagerFactoryClass value null from jpox.properties with org.datanucleus.jdo.JDOPersistenceManagerFactory 12/02/02 15:17:58 DEBUG metastore.ObjectStore: Overriding datanucleus.identifierFactory value null from jpox.properties with datanucleus 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.autoStartMechanismMode = checked 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.identifierFactory = datanucleus 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.transactionIsolation = read-committed 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.validateTables = false 12/02/02 15:17:58 DEBUG metastore.ObjectStore: javax.jdo.option.ConnectionURL = jdbc:mysql://bofa3/metastore 12/02/02 15:17:58 DEBUG metastore.ObjectStore: javax.jdo.option.DetachAllOnCommit = true 12/02/02 15:17:58 DEBUG metastore.ObjectStore: javax.jdo.option.NonTransactionalRead = true 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.fixedDatastore = true 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.validateConstraints = false 12/02/02 15:17:58 DEBUG metastore.ObjectStore: javax.jdo.option.ConnectionDriverName = com.mysql.jdbc.Driver 12/02/02 15:17:58 DEBUG metastore.ObjectStore: javax.jdo.option.ConnectionUserName = hiveuser 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.validateColumns = false 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.cache.level2 = false 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.plugin.pluginRegistryBundleCheck = LOG 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.cache.level2.type = SOFT 12/02/02 15:17:58 DEBUG metastore.ObjectStore: javax.jdo.PersistenceManagerFactoryClass = org.datanucleus.jdo.JDOPersistenceManagerFactory 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.autoCreateSchema = false 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.storeManagerType = rdbms 12/02/02 15:17:58 DEBUG metastore.ObjectStore: datanucleus.connectionPoolingType = DBCP 12/02/02 15:17:58 INFO metastore.ObjectStore: ObjectStore, initialize called FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException 12/02/02 15:17:58 ERROR exec.FetchTask: FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException org.apache.hadoop.hive.ql.metadata.HiveException: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:953) at org.apache.hadoop.hive.ql.exec.DDLTask.showDatabases(DDLTask.java:1668) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:284) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:209) at org.apache.hadoop.hive.cli.CliDriver$processCmd.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) at com.sap.hadoop.windowing.WindowingHiveCliDriver.processCmd(WindowingHiveCliDriver.groovy:148) at com.sap.hadoop.windowing.WindowingHiveCliDriver$processCmd.callCurrent(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) at com.sap.hadoop.windowing.WindowingHiveCliDriver.processLine(WindowingHiveCliDriver.groovy:85) at com.sap.hadoop.windowing.WindowingHiveCliDriver$processLine.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:255) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698) at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:234) at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:261) at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:196) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:171) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:354) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:306) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:451) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:232) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:197) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:108) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:1868) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:1878) at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:951) ... 28 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958) at java.security.AccessController.doPrivileged(Native Method) at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953) at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159) ... 45 more Caused by: java.lang.NullPointerException at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:443) at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:355) at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:215) at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:156) at org.datanucleus.plugin.PluginManager.registerExtensionPoints(PluginManager.java:82) at org.datanucleus.OMFContext.(OMFContext.java:156) at org.datanucleus.OMFContext.(OMFContext.java:137) at org.datanucleus.ObjectManagerFactoryImpl.initialiseOMFContext(ObjectManagerFactoryImpl.java:132) at org.datanucleus.jdo.JDOPersistenceManagerFactory.initialiseProperties(JDOPersistenceManagerFactory.java:363) at org.datanucleus.jdo.JDOPersistenceManagerFactory.(JDOPersistenceManagerFactory.java:307) at org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:255) at org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182) ... 53 more

12/02/02 15:17:58 DEBUG exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:953) at org.apache.hadoop.hive.ql.exec.DDLTask.showDatabases(DDLTask.java:1668) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:284) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:209) at org.apache.hadoop.hive.cli.CliDriver$processCmd.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) at com.sap.hadoop.windowing.WindowingHiveCliDriver.processCmd(WindowingHiveCliDriver.groovy:148) at com.sap.hadoop.windowing.WindowingHiveCliDriver$processCmd.callCurrent(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) at com.sap.hadoop.windowing.WindowingHiveCliDriver.processLine(WindowingHiveCliDriver.groovy:85) at com.sap.hadoop.windowing.WindowingHiveCliDriver$processLine.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:255) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught. NestedThrowables: java.lang.reflect.InvocationTargetException at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803) at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698) at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:234) at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:261) at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:196) at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:171) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:354) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:306) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:451) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:232) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.(HiveMetaStore.java:197) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:108) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:1868) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:1878) at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:951) ... 28 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958) at java.security.AccessController.doPrivileged(Native Method) at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953) at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159) ... 45 more Caused by: java.lang.NullPointerException at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:443) at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:355) at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:215) at org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:156) at org.datanucleus.plugin.PluginManager.registerExtensionPoints(PluginManager.java:82) at org.datanucleus.OMFContext.(OMFContext.java:156) at org.datanucleus.OMFContext.(OMFContext.java:137) at org.datanucleus.ObjectManagerFactoryImpl.initialiseOMFContext(ObjectManagerFactoryImpl.java:132) at org.datanucleus.jdo.JDOPersistenceManagerFactory.initialiseProperties(JDOPersistenceManagerFactory.java:363) at org.datanucleus.jdo.JDOPersistenceManagerFactory.(JDOPersistenceManagerFactory.java:307) at org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:255) at org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182) ... 53 more

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask 12/02/02 15:17:58 ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

hbutani commented 12 years ago

Can you try with version 0.2; this simplifies how integration with Hive: no more spawning of processes, no more interaction with Hive from Map/Reduce tasks... Also contains support for table functions & function chains.

cwsteinbach commented 12 years ago

The NPE in datanucleus is due to HIVE-2473. You may be able to resolve this by selectively removing JARs from your classpath. The other option is to build a one-off version of the datanucleus-core JAR that includes a fix for NUCCORE-689: http://www.datanucleus.org/servlet/jira/browse/NUCCORE-689

hbutani commented 12 years ago

Thanks Carl. Amit can you check if you get this error with version 0.2 Version 0.2 should behave identical to a hive cli process, there is no more spawning of a WindowingEngine process. I am hoping the issue had to do with how the WindowingEngine process was spawned; but if you still get the error then we have to do what Carl suggests.

Just ran the testsuite with Mysql as the metastore, don't see this issue. But I am using hadoop-0.20.2-cdh3u1, and my hive version is from trunk. Have also tested with hive 0.7.1-cdh3u1, but with the configuration that connects via a metastore server that is configured to connect to mysql. I had to do this because my metastore schema was created using trunk. The columns metastore table has changed. If you have a metastore that was created using 0.7.1 you shouldn't need to do this. A direct connection should work.