support for yarn in CHD4.x

[jendap]

ok, feel free to close this one without even thinking about it but ...

Some people (like us) have yarn on CHD4.x in production. Scoobi still does not work with that. But it should be simple to make it run on both. There needs to be some more work done beside commit 5aa655c. Next failing thing is .mapreduce.Job#getTaskCompletionEvents method does not exists with the right types... it should not be hard to support it... on the other hand these things are fixed in hadoop2 GA

For list of possible problems is here: http://hortonworks.com/blog/running-existing-applications-on-hadoop-2-yarn/

[etorreborre]

I don't have lots of time and not even a yarn cluster to test this but here is an attempt to solve the getTaskCompletionEvents issue: 47de450.

[jendap]

I was trying with hadoop-client version 2.0.0-cdh4.4.0. It does not wok since Job does not have JobClient class at all. So calling protected method fails.

Job class contains method

public TaskCompletionEvent[] getTaskCompletionEvents(final int startFrom)

Could we just create JobClient instead of trying to call Job.getJobClient? Looking at Job.runJob and 2.0.0-cdh4.4.0 it begins like this:

  public static RunningJob runJob(JobConf job) throws IOException {
    JobClient jc = new JobClient(job);
    RunningJob rj = jc.submitJob(job);
    ...

The getJobClient in TaksDetailsLogger has instance of job so we can simply create "new JobClient(job)", right?

[jendap]

The commit 47de450 actually break compatibility with both 2.2.0.2.0.6.0-76 (hortonworks) as well as 2.2.0-cdh5.0.0-beta-1 )cloudera) for the same reason (java.lang.NoSuchMethodException: org.apache.hadoop.mapreduce.Job.getJobClient())

BTW: Is it worth fixing? Now when 2.2.x hadoop is GA? I mean for some (like us) it's a blocker for which we can't use scoobi. On the other hand if I'm the only one to notice... We will have 2.2.x GA soon ;-)

[etorreborre]

Sorry about the slow turnaround time on this subject. This week is a very busy one for us. I hope to be able to have a look at that next week (see also here)

[etorreborre]

There is now a SNAPSHOT for yarn support.