search

sequenceiq / hadoop-docker

Hadoop docker image
https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/
Apache License 2.0
1.21k stars 561 forks source link

Example can't connect to ResourceManager #1

Closed markkimsal closed 10 years ago

markkimsal commented 10 years ago

Running the example steps from README.md, when I get to the MR example I get the following output:

14/03/19 19:31:47 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/03/19 19:31:49 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

The last message repeats over and over endlessly. (No files end up in HDFS output/ folder either)

My datanode process is not running, stoping and starting hdfs doesn't launch it.

sbin/stop-dfs.sh 
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: no datanode to stop
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
markkimsal commented 10 years ago

To keep the datanode alive after the namenode -format command I had to source the yarn-env.sh file before formatting the name node

. /usr/local/hadoop/etc/hadoop/yarn-env.sh

I still cannot run the MR example. netstat -ap shows no programs listening to 8032.

markkimsal commented 10 years ago

Ahh, apparently the second sbin/start-dfs.sh should be sbin/start-yarn.sh. I was confused because running format killed my datanode without the yarn environment variables.

New Testing section of README should be:

# start ssh and hdfs
service sshd start
. /usr/local/hadoop/etc/hadoop/hadoop-env.sh
. /usr/local/hadoop/etc/hadoop/yarn-env.sh
cd $HADOOP_HOME
sbin/start-dfs.sh

# format and create directories
bin/hdfs namenode -format
sbin/start-yarn.sh
bin/hdfs dfs -mkdir -p /user/root
bin/hdfs dfs -put etc/hadoop/ input

# run the mapreduce
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar grep input output 'dfs[a-z.]+'

# check the output
bin/hdfs dfs -cat output/*
lalyos commented 10 years ago

Mark thanks for your effort. This work is still in heavy construction. Actually i'm just moving the first section into the image itself. So the image will:

Stay tuned ;)