amplab / docker-scripts

Dockerfiles and scripts for Spark and Shark Docker images
261 stars 102 forks source link

Starting worker container never completed on boot2docker (Docker v1.0.0) #39

Closed dbaba closed 10 years ago

dbaba commented 10 years ago

I was trying to run Spark 1.0.0 on docker with this project scripts. After installing python and bash with tae-ab, I ran the following command on boot2docker vm:

sudo ./deploy/deploy.sh -i amplab/spark:1.0.0 -c

then the command never finished. Did I miss something?

The terminal showed the message waiting for workers to register and dots followed it. The command didn't seem to get stuck as the dots were increased continuously but never finished over a couple of hours.

Here is the entire log until ctrl + c:

** Starting Spark 1.0.0 ***
starting nameserver container
Unable to find image 'amplab/dnsmasq-precise' locally
Pulling repository amplab/dnsmasq-precise

started nameserver container:  8a6c93484ff992538fb7d706cf7d348477920a90f41a6ce7068120c2afb4d04f
DNS host->IP file mapped:      /tmp/dnsdir_25925/0hosts
NAMESERVER_IP:                 172.17.0.2
waiting for nameserver to come up 
starting master container
Unable to find image 'amplab/spark-master:1.0.0' locally
Pulling repository amplab/spark-master

started master container:      009ca3cb6a3bb034af5aaac7f7898d37d7f898d692a1e27cfc28b205982c6575
MASTER_IP:                     172.17.0.3
waiting for master ............
waiting for nameserver to find master 
starting worker container
Unable to find image 'amplab/spark-worker:1.0.0' locally
Pulling repository amplab/spark-worker

started worker container:  ed2ddaefd11e2934cd7ac0d5717f244a64e6997d55791a2b272400642918ffe9
starting worker container
started worker container:  1ffec6ce05937bf37cb96d9ecd3f0cdfd6c66a75c098f964608af144fa3d9f3b
waiting for workers to register ................................................
................................................................................
................................................................................
................................................................................

Any suggestion is appreciated.

dbaba commented 10 years ago

I've found -c argument caused the trouble. -c works only after the initial setup is completed, otherwise the deploy command gets stuck.

First, I ran the following command (without -c):

sudo ./deploy/deploy.sh -i amplab/spark:1.0.0

I could run docker attach with success.

Then I tried the command with -c parameter after killing all spark processes and nameservers,

sudo ./deploy/deploy.sh -i amplab/spark:1.0.0 -c

I finally got Spark worked and received the welcome message.

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.0.0
      /_/