search

weiqingy / caochong

Set up a Hadoop and/or Spark cluster running within Docker containers on a single physical machine
Apache License 2.0
78 stars 35 forks source link

run.sh from-source fails because the ssh service hasn't started in the docker container #12

Open anuvab opened 7 years ago

anuvab commented 7 years ago

Running this script (caochong/from-source/run.sh) gives this error:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [caochong-master]
caochong-master: ssh: connect to host caochong-master port 22: Connection refused
localhost: ssh: connect to host localhost port 22: Connection refused
Starting secondary namenodes [0.0.0.0]
0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused
17/01/16 06:51:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
chown: missing operand after '/hadoop/logs'
Try 'chown --help' for more information.
starting resourcemanager, logging to /hadoop/logs/yarn--resourcemanager-fbad3ebe7cdf.out
localhost: ssh: connect to host localhost port 22: Connection refused
starting resourcemanager, logging to /hadoop/logs/yarn--resourcemanager-ed26fb8707b6.out

The relevant part is

localhost: ssh: connect to host localhost port 22: Connection refused

The reason is that the ssh service hasn't been started in the docker container.

The simplest fix for this would be to add a line to start ssh in the container,i.e add this line:

# Start hdfs and yarn services
docker exec -it $master_id service ssh start
docker exec -it $master_id $HADOOP_HOME/sbin/start-dfs.sh
docker exec -it $master_id $HADOOP_HOME/sbin/start-yarn.sh

I have tried this solution and it works well. Am I missing something, or should this line be added to the script?

liuml07 commented 7 years ago

We use the supervisor to start the sshd service (and other incoming services) automatically; it must stop working sometimes. Will have a look at this.