suvodeep-pyne
commented
9 years ago @shengli
You are using a multimap. This kind of performs an implicit UNION. The final output from each multimap should have the same name for it to compile. Also, you are missing a Reduce Phase and you have 5 reducers. Plus you need to mention the SHUFFLE phase as well. However, in this case I think you are trying to do a join here? That can be done in a single MAP block.
I haven't tested the following code give it a shot and let me know if that works.
JOB "Join count words" REDUCERS 5; MAP { a = LOAD "/cubert/words.txt" USING TEXT("schema": "STRING word");
b = LOAD-CACHED "/cubert/words.txt" USING TEXT("schema": "STRING word");
test_joined = HASH-JOIN a BY word, b BY word;
} STORE test_joined INTO "/cubert/woud_count/join_output" USING TEXT(); END
Regards Suvodeep
On Tue, Jun 23, 2015 at 1:11 AM, Sheng, Li notifications@github.com wrote:
Hi, @mparkhe https://github.com/mparkhe Would you like to give an example for a join operator. I follow the document in http://linkedin.github.io/Cubert/operators/join.html but seems not work.
JOB "Join count words" REDUCERS 5; MAP { a = LOAD "/cubert/words.txt" USING TEXT("schema": "STRING word"); }
MAP { b = LOAD "/cubert/words.txt" USING TEXT("schema": "STRING word"); } test_joined = HASH-JOIN a BY word, b BY word; STORE test_joined INTO "/cubert/woud_count/join_output" USING TEXT();END
shengli-mac$ cubert join.cmr Using HADOOP_CLASSPATH=:/Users/shengli/git_repos/cubert/release/lib/* line 13:9 mismatched input '=' expecting {'.', ID} line 13:23 mismatched input 'BY' expecting '{'
Cannot parse cubert script. Exiting. PROGRAM "Join Word Count";
Cannot compile cubert script. Exiting. Exception in thread "main" java.text.ParseException at com.linkedin.cubert.plan.physical.PhysicalParser.parsingTask(PhysicalParser.java:197) at com.linkedin.cubert.plan.physical.PhysicalParser.parseInputStream(PhysicalParser.java:161) at com.linkedin.cubert.plan.physical.PhysicalParser.parseProgram(PhysicalParser.java:156) at com.linkedin.cubert.ScriptExecutor.compile(ScriptExecutor.java:304) at com.linkedin.cubert.ScriptExecutor.main(ScriptExecutor.java:523) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
— Reply to this email directly or view it on GitHub https://github.com/linkedin/Cubert/issues/7.
OopsOutOfMemory
Hi, @mparkhe Would you like to give an example for a join operator. I follow the document in http://linkedin.github.io/Cubert/operators/join.html but seems not work.
shengli-mac$ cubert join.cmr Using HADOOP_CLASSPATH=:/Users/shengli/git_repos/cubert/release/lib/* line 13:9 mismatched input '=' expecting {'.', ID} line 13:23 mismatched input 'BY' expecting '{'
Cannot parse cubert script. Exiting. PROGRAM "Join Word Count";
Cannot compile cubert script. Exiting. Exception in thread "main" java.text.ParseException at com.linkedin.cubert.plan.physical.PhysicalParser.parsingTask(PhysicalParser.java:197) at com.linkedin.cubert.plan.physical.PhysicalParser.parseInputStream(PhysicalParser.java:161) at com.linkedin.cubert.plan.physical.PhysicalParser.parseProgram(PhysicalParser.java:156) at com.linkedin.cubert.ScriptExecutor.compile(ScriptExecutor.java:304) at com.linkedin.cubert.ScriptExecutor.main(ScriptExecutor.java:523) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)