search

mesos / hadoop

Hadoop on Mesos
176 stars 80 forks source link

Hadoop on Mesos uses only one node? #63

Closed echupriyanov closed 9 years ago

echupriyanov commented 9 years ago

I’m having an issue running Hadoop job on Mesos cluster. I have followed README and was suceccsful starting JobTracker and running wordcount example Hadoop job on cluster.

However, when I try to launch a larger job (Camus export data from Kafka to HDFS) I see only one TaskTracker started `allocating only 2 Map slots (default configured for a node) and not using any other nodes (cluster consists of 5 nodes and I requested total 30 Map tasks).

In my setup I use Cloudera Hadoop distribution version 2.6.0-cdh5.4.2 Mesos version 0.22.1 And latest mesos-hadoop-mr1-0.1.1 (git commit c972174cedcf2d07d35207297de398addddf6bfc)

What am I missing? Or is it intended behavior? Thanks.

tarnfeld commented 9 years ago

Does your slave have enough resources to accommodate more than two slots? The resources required for a TaskTracker are defined here and then any remaining resources will be cut up into as many map/reduce slots as possible, or the number of pending slots, whichever is less.

You might need to play with the resource allocation in your cluster to get things working smoothly. If you're still having trouble, could you share more details about your configuration?

Also, check that you have a ports resource on your mesos slaves, ports are required for the Hadoop on Mesos framework to accept offers.

tarnfeld commented 9 years ago

@echupriyanov It'd be great if you could let me know if you found the cause? I'm going to close this ticket for now...