Improvement for MesosScheduler, cleanups and lint removal

briantopping commented 9 years ago

In my deployment of 2.6.0-mr1-cdh5.4.0, I had a problem where "2015-05-08 03:33:24,421 INFO org.apache.hadoop.mapred.MesosScheduler: Unknown/exited TaskTracker: http://10.211.55.16:50060. " was being emitted 3-5 times a second. I don't have a JWDP connection figured out to see what the messages were, but I'm guessing there is an internal housekeeping job that is used internally. This implementation allows that to happen quietly.

Restructure o.a.h.mapred.MesosScheduler#assignTasks to always pass through tasks, then decide whether they are a Mesos-managed TaskTracker
ResourcePolicy is abstract for all intents, but it could be instantiated. Make it literally abstract
A bunch of lint warnings removed
Rearrange code to be easier to read -- interface implementations commented and methods in order, update some docs to JavaDoc format.
Update POM version to 0.1.1-SNAPSHOT for additional work.

tarnfeld commented 9 years ago

Thanks for the contributions! Nice to see the code being tidied up. I've made a couple of comments and once those are done i'll merge it and tag a release.

briantopping commented 9 years ago

Thanks for the time to review this so quickly! It's always a pleasure to contribute when PRs don't go stale. :)

Please give me a bit to get the release plugin set up. I'll document how to use it and we can try a release from there, sound good?

tarnfeld commented 9 years ago

Sounds great!

briantopping commented 9 years ago

Can I go ahead and set up artifact publishing to Sonatype Maven repository? This isn't an official Apache project, so we can't release it there.

tarnfeld commented 9 years ago

I wonder if the folk over at Mesosphere would be kind enough to host the project in their repos? They have a public maven repo over at http://downloads.mesosphere.io/maven. I'll ask around!

briantopping commented 9 years ago

Oh wait, my bad:

    <groupId>org.apache.mesos</groupId>
    <artifactId>hadoop-mesos</artifactId>
    <version>0.1.1-SNAPSHOT</version>

I can get this into the Apache repo if that's okay with the PMC.

tarnfeld commented 9 years ago

Maybe send an email to the dev mailing list, or ping the IRC? #mesos

briantopping commented 9 years ago

Pushing directly to Sonatype is the generally accepted clean path for small projects. If it's an Apache project, it should be pushed to the Apache repos. Both go directly to Maven Central.

If this isn't an Apache project, we should use a different GAV (group/artifact/version) combo.

WDYT?

tarnfeld commented 9 years ago

I'm not sure if this is now technically an Apache project. It probably has that GAV because it was originally part of the Mesos source code and was brought out into it's own project by @brndnmtthws some time ago.

I think sending a message to the mailing list would be worth it, there's a bit of a wider discussion around this anyway related to hosting frameworks under /mesos on github (and if we're continuing to do that, my vote goes for hosting it in the Apache repos).

briantopping commented 9 years ago

Sent: http://www.mail-archive.com/dev@mesos.apache.org/msg32398.html

tarnfeld commented 9 years ago

Awesome. How would you feel about us taking this release configuration into a new PR and just keeping this one for the code cleanups/fixes?

briantopping commented 9 years ago

Sounds like an excellent idea. I've rolled back the version change per your request.

mesos / hadoop

Improvement for MesosScheduler, cleanups and lint removal #55