search

mesos / hadoop

Hadoop on Mesos
176 stars 80 forks source link

Need help in configuring hadoop on mesos cluster #72

Open viswram opened 8 years ago

viswram commented 8 years ago

My requirement is to configure hadoop on mesos where mesos and hadoop will be installed on different servers. I have below queries on that 1) Do I need to configure hadoop on the same server as mesos master server or can they be on different servers? (my requirement is to configure hadoop & mesos on different servers) 2) Is it necessary to configure the hadoop first before proceeding with the steps on this link https://github.com/mesos/hadoop.

I am a storage guy and hadoop/mesos are totally new to me. Any help on this with basic information would be really appreciated. Thanks.

DarinJ commented 8 years ago

viswram answers to your questions: 1) no it can be on a completely different server, I've even run it in marathon (though it requires knowledge of service discovery, so do do that at the start). 2) It's expected HDFS is setup in some form already (consider https://github.com/mesosphere/hdfs).

Shameless plug: You my also want to look at myriad.apache.org which is YARN running on Mesos vs just Map/Reduce.

viswram commented 8 years ago

Thanks Darin J. 1) I believe hadoop and HDFS setup are one and the same because one article says hadoop (https://strat0sphere.wordpress.com/2014/10/30/hadoop-on-mesos-installation-guide/) and the other says HDFS (https://strat0sphere.wordpress.com/2014/10/30/cloudera-hdfs-cdh5-installation-to-use-with-mesos/). I am bit confused here with these terminologies as I am completely new to this technology. 2) Is it required to keep the hadoop master as a part of mesos cluster? Apologies for these silly questions. Thanks.

DarinJ commented 8 years ago

1) That is incorrect - You'll need to setup HDFS separately, i.e, Namenodes, Secondary Namenodes and datanodes. The mesos-hadoop Framework only sets up the Jobtracker and tasktrackers. 2) I'm not sure what you mean by hadoop master - the JobTracker? Technically no the job tracker doesn't need to be part of the mesos cluster but will need to be able to communicate with the mesos-master (to register and receive offers) and the mesos-slaves (to run mapreduce jobs).

viswram commented 8 years ago

Hi Darin, I am getting some idea now. Let me explain you my setup now. Server A - This has been installed with mesos and it is a mesos master. Server B - This is the hadoop server. This server is not part of mesos cluster. I have installed hadoop using this procedure (https://strat0sphere.wordpress.com/2014/10/30/hadoop-on-mesos-installation-guide/) and "libmesos.so" library file is required to proceed further but as mesos is not installed on this server I couldn't find this file on this server. So I am stuck half way here. Am I following the correct procedure or did I misss something? Do I need to install or configure any hadoop related packages on Server A? Please guide me. Appreciate your support.