search

infochimps-labs / wonderdog

Bulk loading for elastic search
http://infochimps.com
Apache License 2.0
186 stars 56 forks source link

Version Incompatibility? #13

Open alexmc6 opened 11 years ago

alexmc6 commented 11 years ago

When running a Wonderdog job in the current CDH 4.1.1 environment I get...

Backend error message

Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected

Pig Stack Trace

ERROR 2997: Unable to recreate exception from backed error: Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected

org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backed error: Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:217) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:149) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:383) at org.apache.pig.PigServer.launchPlan(PigServer.java:1275) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1260) at org.apache.pig.PigServer.execute(PigServer.java:1250) at org.apache.pig.PigServer.executeBatch(PigServer.java:362) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:132) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84) at org.apache.pig.Main.run(Main.java:555) at org.apache.pig.Main.main(Main.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Is this due to maven's pom.xml saying

<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-core</artifactId>
  <version>0.20.2</version>
</dependency>

<dependency>
  <groupId>org.apache.pig</groupId>
  <artifactId>pig</artifactId>
  <version>0.8.0</version>
</dependency>

when those two are no longer right?

PS

$ hadoop version Hadoop 2.0.0-cdh4.1.1 Subversion file:///var/lib/jenkins/workspace/generic-package-ubuntu64-12-04/CDH4.1.1-Packaging-Hadoop-2012-10-16_10-36-39/hadoop-2.0.0+545-1.cdh4.1.1.p0.5~precise/src/hadoop-common-project/hadoop-common -r 581959ba23e4af85afd8db98b7687662fe9c5f20 Compiled by jenkins on Tue Oct 16 11:14:06 PDT 2012 From source with checksum 95f5c7f30b4030f1f327758e7b2bd61f

$ pig -version Apache Pig version 0.10.0-cdh4.1.1 (rexported) compiled Oct 16 2012, 12:27:24

mrflip commented 11 years ago

I am fairly sure this will not work with the new "MR2" (yarn) mapreduce API in the 2.x branch of Hadoop (as featured in CDH4).

I won't be surprised if it doesn't work even with the "MR1" (backwards-compatible) API in the 2.x branch of Hadoop, but I think it should be a shorter gap to close.

Are you launching against MR2 or MR1?