cluster doesn't start

[amtrack]

When building the docker image from source as described in the README no single node of the cluster is starting.

$ docker -v
Docker version 0.9.0, build 2b3fdf2

$ DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING=1 DOCKER_RIAK_CS_CLUSTER_SIZE=5 DOCKER_RIAK_CS_DEBUG=1 make start-cluster
./bin/start-cluster.sh
+ DOCKER_RIAK_CS_CLUSTER_SIZE=5
+ egrep hectcastro/riak
+ docker ps
+ echo

+ echo 'Bringing up cluster nodes:'
Bringing up cluster nodes:
+ echo

++ seq -f %02g 1 5
+ for index in '$(seq -f "%02g" "1" "${DOCKER_RIAK_CS_CLUSTER_SIZE}")'
+ '[' 01 -gt 1 ']'
+ docker run -e DOCKER_RIAK_CS_CLUSTER_SIZE=5 -e DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING=1 -P --name riak-cs01 -d hectcastro/riak-cs
++ egrep 'riak-cs01[^/]'
++ docker ps
++ cut '-d ' -f1
+ CONTAINER_ID=cca37c789ab6
++ cut -d : -f2
++ docker port cca37c789ab6 8080
+ CONTAINER_PORT=49230
+ egrep OK
+ curl -s http://localhost:49230/riak-cs/ping
+ sleep 3
+ egrep OK
+ curl -s http://localhost:49230/riak-cs/ping
+ sleep 3
+ egrep OK
+ curl -s http://localhost:49230/riak-cs/ping
+ sleep 3

This goes on for at least 15 minutes.

Starting the first node manually in attached mode will result in this:

$ docker run -e "DOCKER_RIAK_CS_CLUSTER_SIZE=5" -e "DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING=1" -P --name "riak-cs01" hectcastro/riak-cs
No SSH host key available. Generating one...
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...
Node is not running!

And with cluster size 1:

$ docker run -e "DOCKER_RIAK_CS_CLUSTER_SIZE=1" -P --name "riak-cs01" hectcastro/riak-cs
No SSH host key available. Generating one...
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...

Same thing happens by the way when trying to build and start a riak cluster. Using the image from index.docker.io seems to work.

[hectcastro]

First of all, thanks for taking this project for a spin.

Can you telnet to the ports within the curl command? Where is your Docker daemon running?

[amtrack]

I'm running both the docker daemon and the client on a VPS with Ubuntu 13.10. So my DOCKER_HOST environment variable is unset.

$ curl http://localhost:49242/riak-cs/ping
curl: (52) Empty reply from server
$ telnet localhost 49242
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
Connection closed by foreign host.

and the running riak-cs docker containers:

$ docker ps | grep riak-cs
7404ad8fc4c5        hectcastro/riak-cs:latest           /sbin/my_init --quie   2 minutes ago       Up 2 minutes        0.0.0.0:49242->8080/tcp, 0.0.0.0:49243->8098/tcp   riak-cs01

Trying to connect via the container IP address:

$ docker inspect riak-cs01 | grep IPAddress
        "IPAddress": "172.17.0.11",
$ telnet 172.17.0.11 8080
Trying 172.17.0.11...
telnet: Unable to connect to remote host: Connection refused
$ telnet 172.17.0.11 8098
Trying 172.17.0.11...
telnet: Unable to connect to remote host: Connection refused

And the log of the container again:

$ docker logs riak-cs01
No SSH host key available. Generating one...
Creating SSH2 RSA key; this may take some time ...
Creating SSH2 DSA key; this may take some time ...
Creating SSH2 ECDSA key; this may take some time ...
Node is not running!

[amtrack]

@hectcastro: Just to let you know, it works fine with boot2docker v0.8.0 and docker 0.10 So i think is maybe a docker 0.9 related issue. I'll investigate this further...

[hectcastro]

Interesting. I will try to reproduce locally on 0.9. At the very least this can lead to a warning in the README.

[hectcastro]

I was able to reproduce this issue with Docker client/server v0.9.0 on Ubuntu 13.10. The following changes to start-cluster.sh allowed me to bring the cluster up:

diff --git a/bin/start-cluster.sh b/bin/start-cluster.sh
index 8ca425b..0412f5f 100755
--- a/bin/start-cluster.sh
+++ b/bin/start-cluster.sh
@@ -27,12 +27,12 @@ echo
 for index in $(seq -f "%02g" "1" "${DOCKER_RIAK_CS_CLUSTER_SIZE}");
 do
   if [ "${index}" -gt "1" ] ; then
-    docker run -e "DOCKER_RIAK_CS_CLUSTER_SIZE=${DOCKER_RIAK_CS_CLUSTER_SIZE}" \
+    docker run -t -e "DOCKER_RIAK_CS_CLUSTER_SIZE=${DOCKER_RIAK_CS_CLUSTER_SIZE}" \
                -e "DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING=${DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING}" \
                -P --name "riak-cs${index}" --link "riak-cs01:seed" \
                -d hectcastro/riak-cs > /dev/null 2>&1
   else
-    docker run -e "DOCKER_RIAK_CS_CLUSTER_SIZE=${DOCKER_RIAK_CS_CLUSTER_SIZE}" \
+    docker run -t -e "DOCKER_RIAK_CS_CLUSTER_SIZE=${DOCKER_RIAK_CS_CLUSTER_SIZE}" \
                -e "DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING=${DOCKER_RIAK_CS_AUTOMATIC_CLUSTERING}" \
                -P --name "riak-cs${index}" -d hectcastro/riak-cs > /dev/null 2>&1
   fi

This appears to be related to https://github.com/phusion/baseimage-docker/issues/36 and https://github.com/dotcloud/docker/issues/4605.

Instead of altering start-cluster.sh, I'd rather add a note to the README requiring Docker 0.10.0.

[hectcastro]

Included a note requiring Docker 0.10.0 in 42471a35f68d19b7cef7768f110489dfabb20134.

[amtrack]

Awesome. Thank you!