search

mesos / hadoop

Hadoop on Mesos
176 stars 80 forks source link

Split MAP and REDUCE tasks into individual mesos tasks #60

Open tarnfeld opened 9 years ago

tarnfeld commented 9 years ago

This commit splits out the resources for MAP and REDUCE slots into two Mesos tasks instead of one, while still using a single TaskTracker JVM. This allows the idle-slot tracking to operator on MAP and REDUCE slots individually further increasing our ability to release idle resources faster.

This is an implementation of #47.

tarnfeld commented 9 years ago

image

This screenshot was taken while a job was running on a shared cluster, and it's possible to see quite clearly that some Reduce slots (from the 0th task tracker) were revoked and the resources freed, while the maps are still allocated.

brndnmtthws commented 9 years ago

Looks pretty good to me. If you've tested it, merge away.

Might be worth bumping the version in the pom too.

tarnfeld commented 9 years ago

Thanks for the quick review! I've just rolled this out on one of our clusters so I want to let things settle a bit first, and get some serious traffic through the JT/TTs before saying it's good to go.