search

hbutani / SQLWindowing

SQL Windowing Functions for Hadoop
65 stars 17 forks source link

mismatched input '/tmp/hadhive/' #20

Closed penghao122 closed 12 years ago

penghao122 commented 12 years ago

is ther anyone to encountered below issue? Any help or point are appreciated. thx. hive> from

<select starttime,lac ,ci ,callerno,count(1) callCount from ranap group by starttime,lac ,ci ,callerno limit 1000

partition by lac order by lac with rank() as r select starttime,lac,ci,callerno,callCount,r where <r <=3> into path '/tmp/hadhive/' serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' with serdeproperties('field.delim'=',') format 'org.apache.hadoop.mapred.TextOutputFormat'; BR.recoverFromMismatchedToken com.sap.hadoop.windowing.WindowingException: line 12:10 mismatched input '/tmp/hadhive/' expecting EQ at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190) at com.sap.hadoop.windowing.runtime.WindowingShell.parse(WindowingShell.groovy:72) at com.sap.hadoop.windowing.runtime.WindowingShell.execute(WindowingShell.groovy:126) at com.sap.hadoop.windowing.runtime.WindowingShell$execute.call(Unknown Source) at com.sap.hadoop.windowing.cli.WindowingClient3.executeQuery(WindowingClient3.groovy:28) at com.sap.hadoop.windowing.cli.WindowingClient3$executeQuery.call(Unknown Source) at com.sap.hadoop.windowing.WindowingHiveCliDriver.processCmd(WindowingHiveCliDriver.groovy:117) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341) at org.apache.hadoop.hive.cli.CliDriver$processLine.call(Unknown Source) at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:235) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

hbutani commented 12 years ago

Looks like you are missing a '<'

from <from ranap group by starttime,lac ,ci ,callerno limit 1000

partition by lac order by lac with rank() as r select starttime,lac,ci,callerno,callCount,r where into path '/tmp/hadhive/' serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' with serdeproperties('field.delim'=',') format 'org.apache.hadoop.mapred.TextOutputFormat';

penghao122 commented 12 years ago

not missing a '<' can u check below log

hive> from

<select starttime,lac ,ci ,callerno,count(1) callCount from ranap group by starttime,lac ,ci ,callerno limit 1000

partition by lac order by lac with rank() as r select starttime,lac,ci,callerno,callCount,r where <r <=3> into path 'tmp/test' serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' with serdeproperties('field.delim'=',') format 'org.apache.hadoop.mapred.TextOutputFormat'; BR.recoverFromMismatchedToken com.sap.hadoop.windowing.WindowingException: line 12:10 mismatched input 'tmp/test' expecting EQ at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190) at com.sap.hadoop.windowing.runtime.WindowingShell.parse(WindowingShell.groovy:72) at com.sap.hadoop.windowing.runtime.WindowingShell.execute(WindowingShell.groovy:126) at com.sap.hadoop.windowing.runtime.WindowingShell$execute.call(Unknown Source) at com.sap.hadoop.windowing.cli.WindowingClient3.executeQuery(WindowingClient3.groovy:28) at com.sap.hadoop.windowing.cli.WindowingClient3$executeQuery.call(Unknown Source) at com.sap.hadoop.windowing.WindowingHiveCliDriver.processCmd(WindowingHiveCliDriver.groovy:117) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341) at org.apache.hadoop.hive.cli.CliDriver$processLine.call(Unknown Source) at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:235) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Failed windowing query from <select starttime,lac ,ci ,callerno,count(1) callCount from ranap group by starttime,lac ,ci ,callerno limit 1000

partition by lac order by lac with rank() as r select starttime,lac,ci,callerno,callCount,r where <r <=3> into path 'tmp/test' serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' with serdeproperties('field.delim'=',') format 'org.apache.hadoop.mapred.TextOutputFormat' line 12:10 mismatched input 'tmp/test' expecting EQ hive>

penghao122 commented 12 years ago

may the website display issue , i use below sql, it raise above issue

from <select starttime,lac ,ci ,callerno,count(1) callCount from ranap group by starttime,lac ,ci ,callerno limit 1000

partition by lac order by lac with rank() as r select starttime,lac,ci,callerno,callCount,r where <r <=3> into path 'tmp/test' serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' with serdeproperties('field.delim'=',') format 'org.apache.hadoop.mapred.TextOutputFormat';

penghao122 commented 12 years ago

it's the key .so .if have no privlidge to, then raise these error message

BR.recoverFromMismatchedToken com.sap.hadoop.windowing.WindowingException: line 12:10 mismatched input '/tmp/hadhive/' expecting EQ

penghao122 commented 12 years ago

when i deleted the big db , and then try to the same sql, it raise another issue

hive> wmode windowing; hive> from

<select starttime,lac ,ci ,callerno from bssap group by starttime,lac ,ci ,callerno limit 1000> partition by lac order by lac with rank() as r, count(callerno) as callCount select starttime,lac,ci,callerno,r,callCount where <r <=3>; Executing Embedded Hive Query: Create table WindowingTempTable_1345169160306 as select starttime,lac ,ci ,callerno from bssap group by starttime,lac ,ci ,callerno limit 1000

Total MapReduce jobs = 2 Launching Job 1 out of 2 Number of reduce tasks not specified. Defaulting to jobconf value of: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer= In order to limit the maximum number of reducers: set hive.exec.reducers.max= In order to set a constant number of reducers: set mapred.reduce.tasks= Cannot run job locally: Input Size (= 477931606) is larger than hive.exec.mode.local.auto.inputbytes.max (= 134217728) Starting Job = job_201208161534_0108, Tracking URL = http://server52:50030/jobdetails.jsp?jobid=job_201208161534_0108 Kill Command = /home/hadhive/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=server52:9001 -kill job_201208161534_0108 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2012-08-17 10:06:04,956 Stage-1 map = 0%, reduce = 0% 2012-08-17 10:06:29,029 Stage-1 map = 100%, reduce = 0% 2012-08-17 10:06:41,068 Stage-1 map = 100%, reduce = 70% 2012-08-17 10:06:44,079 Stage-1 map = 100%, reduce = 100% Ended Job = job_201208161534_0108 Launching Job 2 out of 2 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer= In order to limit the maximum number of reducers: set hive.exec.reducers.max= In order to set a constant number of reducers: set mapred.reduce.tasks= Selecting local mode for task: Stage-2 Execution log at: /tmp/hadhive/hadhive_20120817100606_4fffa47b-99a9-40a8-9626-3f1c376e63e5.log Job running in-process (local Hadoop) Hadoop job information for null: number of mappers: 0; number of reducers: 0 2012-08-17 10:06:51,117 null map = 100%, reduce = 0% 2012-08-17 10:06:52,118 null map = 100%, reduce = 100% Ended Job = job_local_0001 Execution completed successfully Mapred Local Task Succeeded . Convert the Join into MapJoin Moving data to: hdfs://server52:9000/smp/hive/warehouse/windowingtemptable_1345169160306 Table default.windowingtemptable_1345169160306 stats: [num_partitions: 0, num_files: 1, num_rows: 1000, total_size: 32175, raw_data_size: 31175] MapReduce Jobs Launched: Job 0: Map: 1 Reduce: 1 HDFS Read: 477931606 HDFS Write: 43691 SUCCESS Total MapReduce CPU Time Spent: 0 msec OK Time taken: 52.573 seconds Finished Executing Embedded Hive Query

com.sap.hadoop.windowing.WindowingException: com.sap.hadoop.windowing.WindowingException: Query doesn't contain an output Path for results at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190) at com.sap.hadoop.windowing.query.Translator.setupOutput(Translator.groovy:348) at com.sap.hadoop.windowing.query.Translator$setupOutput.callCurrent(Unknown Source) at com.sap.hadoop.windowing.query.Translator.translate(Translator.groovy:52) at com.sap.hadoop.windowing.query.Translator$translate.call(Unknown Source) at com.sap.hadoop.windowing.runtime.WindowingShell.execute(WindowingShell.groovy:127) at com.sap.hadoop.windowing.runtime.WindowingShell$execute.call(Unknown Source) at com.sap.hadoop.windowing.cli.WindowingClient3.executeQuery(WindowingClient3.groovy:28) at com.sap.hadoop.windowing.cli.WindowingClient3$executeQuery.call(Unknown Source) at com.sap.hadoop.windowing.WindowingHiveCliDriver.processCmd(WindowingHiveCliDriver.groovy:117) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341) at org.apache.hadoop.hive.cli.CliDriver$processLine.call(Unknown Source) at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:235) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: com.sap.hadoop.windowing.WindowingException: Query doesn't contain an output Path for results at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77) at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:54) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190) at com.sap.hadoop.windowing.runtime.mr.MRTranslator.validateOutputSpec(MRTranslator.groovy:70) at com.sap.hadoop.windowing.runtime.mr.MRTranslator$validateOutputSpec.callCurrent(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141) at com.sap.hadoop.windowing.query.Translator.setupOutput(Translator.groovy:320) ... 17 more Failed windowing query from <select starttime,lac ,ci ,callerno from bssap group by starttime,lac ,ci ,callerno limit 1000> partition by lac order by lac with rank() as r, count(callerno) as callCount select starttime,lac,ci,callerno,r,callCount where <r <=3> com.sap.hadoop.windowing.WindowingException: Query doesn't contain an output Path for results hive>

hbutani commented 12 years ago

I should have read the error message more carefully. As it says 12:10 mismatched input 'tmp/test' expecting EQ

You need an '=' sign after path. So try the query:

from <from ranap group by starttime,lac ,ci ,callerno limit 1000

partition by lac order by lac with rank() as r select starttime,lac,ci,callerno,callCount,r where into path='/tmp/hadhive/' serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' with serdeproperties('field.delim'=',') format 'org.apache.hadoop.mapred.TextOutputFormat';

penghao122 commented 12 years ago

ues .missing the '=' sing