search

dnordgren / CSCE835

Workspace for Group 8 of CSCE 435/835 (Cluster & Grid Computing)
head.pi.cluster.unl.edu :)
0 stars 0 forks source link

Install and configure MapReduce #15

Closed bhushit closed 9 years ago

bhushit commented 9 years ago

Install openjdk

sudo yum install java-1.7.0-openjdk-devel

Install jobtracker on head,

sudo yum install hadoop-0.20-mapreduce-jobtracker
bhushit commented 9 years ago

Create /tmp dir in hdfs as hdfs user,

hdfs dfs -mkdir /tmp
sudo -u hdfs hdfs dfs -chown mapred /tmp
sudo -u hdfs hdfs dfs -chgrp mapred /tmp
bhushit commented 9 years ago

Create config for MapReduce,

sudo touch /etc/hadoop/conf/mapred-site.xml
sudo vim /etc/hadoop/conf/mapred-site.xml

and add the config,

<configuration>
 <property>
 <name>mapred.job.tracker</name>
 <value>hcc-group8head.unl.edu:54311</value>
 </property>
</configuration>
bhushit commented 9 years ago

On workers: Install openjdk

sudo yum install java-1.7.0-openjdk-devel

Install tasktracker on worker,

sudo yum install hadoop-0.20-mapreduce-tasktracker

Create the config file, /etc/hadoop/conf/mapred-site.xml,

<configuration>
 <property>
 <name>mapred.job.tracker</name>
 <value>hcc-group8head.unl.edu:54311</value>
 </property>
</configuration>

Start the task tracker,

sudo /etc/init.d/hadoop-0.20-mapreduce-tasktracker start

Logs will be stored at, /var/log/hadoop-0.20-mapreduce/hadoop-hadoop-tasktracker-work1.cluster8.unl.edu.log

bhushit commented 9 years ago

Changing filename from mapred-default.xml to mapred-site.xml made the jobtracker work We can check the jobtracker status here.