swarm with powerstrip and docker ps vs docker ps -a

[aanm]

I'm trying to run compose with the following architecture:

          docker client (compose)

                |

           swarm daemon

           /          \

       node1          node2

         |              |

    powerstrip       powerstrip

        |               |

      docker          docker

Unfortunately this happens:

$ DOCKER_HOST=localhost:2373 docker ps --filter "name=compose"
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
$ #Trying with -a
$ DOCKER_HOST=localhost:2373 docker ps --filter "name=compose" -a
CONTAINER ID        IMAGE                COMMAND                CREATED             STATUS              PORTS                         NAMES
35cf88f3afb1        compose_web:latest   "python app.py"        12 minutes ago                          192.168.50.6:5000->5000/tcp   ubuntu-14/compose_web_1     
2f3a3933c45b        redis:latest         "/entrypoint.sh redi   12 minutes ago                          6379/tcp                      ubuntu-14/compose_redis_1

Now, I have another swarm master running at the same time with the same architecture above without connecting to powerstrip but connecting directly to docker daemon.

          docker client (compose)

                |

           swarm daemon

           /          \

       node1          node2

        |               |

      docker          docker

$ DOCKER_HOST=localhost:2374 docker ps --filter "name=compose"
CONTAINER ID        IMAGE                COMMAND                CREATED             STATUS              PORTS                         NAMES
35cf88f3afb1        compose_web:latest   "python app.py"        13 minutes ago      Up 13 minutes       192.168.50.6:5000->5000/tcp   ubuntu-14/compose_web_1     
2f3a3933c45b        redis:latest         "/entrypoint.sh redi   13 minutes ago      Up 13 minutes       6379/tcp                      ubuntu-14/compose_redis_1 

The remarkable part is the lack of "status" between the 2.

I'm running on both swarm nodes:

$ docker run -d --name powerstrip-debug \
    --expose 80 \
    binocarlos/powerstrip-debug
$ docker run -d --name powerstrip \
         -v /home/vagrant/adapters-local.yml:/etc/powerstrip/adapters.yml \
         -v /var/run/docker.sock:/var/run/docker.sock \
         --link powerstrip-debug:debug \
         -p 2375:2375 \
         clusterhq/powerstrip:v0.0.1

and the content of /home/vagrant/adapters.yml is:

version: 1
endpoints:
  "GET /*/containers":
    pre: [debug]
  "GET /*/containers":
    post: [debug]
adapters:
  debug: http://debug/v1/extension

 $ docker logs powerstrip-debug
server listening on port: 80

[binocarlos]

@aanm It's interesting that there is no output from the debug container - I would expect to see logging from a request to GET /v1.15/containers as the output from docker logs powerstrip-debug

Fundamentally there is nothing that your powerstrip setup is doing that should affect the requests to GET /*/containers so it seems that something odd is going on.

What happens when you DOCKER_HOST=localhost:2373 docker ps? (i.e. go via Powerstrip but without --filter)

[aanm]

@binocarlos without --filter the behavior is the same. I have noticed, via wireshark, that when I run DOCKER_HOST=localhost:2373 docker ps I see a packet going out from my machine and going to swarm master (which is also running on my machine) and then I do not see any packets coming out from swarm master to the nodes, only the answer for that docker ps command. I have realized though, that I see packets going out of swarm master every +- 30 seconds polling the nodes with containers?getall=1 or something similar. Well, I've also tried to do DOCKER_HOST=192.168.50.6:2375 docker ps (it's the direct address to node 2 "powerstriped". and I see everything

$ DOCKER_HOST=192.168.50.6:2375 docker ps
CONTAINER ID        IMAGE                         COMMAND                CREATED             STATUS              PORTS                                            NAMES
ce86c2f6431b        compose_web:latest            "python app.py"        8 hours ago         Up 8 hours                                                           compose_web_1              
2f050cd9c305        redis:latest                  "/entrypoint.sh redi   8 hours ago         Up 8 hours                                                           compose_redis_1  

So I believe the problem is, after create powerstrip doesn't return something that swarm master or compose is expecting.

$ DOCKER_HOST=localhost:2373 docker-compose up
Attaching to #[yep, it's empty]

$ DOCKER_HOST=localhost:2374 docker-compose up 
Attaching to compose_redis_1

Can you try replicated the issue? I'm using last version (from git) of swarm and compose. Can you try with the stable versions of both?

[aanm]

@binocarlos root problem:

• docker swarm needs to listen for events.
• clusterhq/powerstrip:v0.0.1 doesn't support events.
• clusterhq/powerstrip:latest supports events.

Since it's on README that there's a v0.0.2 can you tag it on dockerhub? I only found v0.0.1 and latest.

[binocarlos]

@aanm great work finding the problem - yes this now makes sense.

With powerstrip v0.0.1 - calls to docker attach or any kind of long lived TCP connection to Docker fails. This happens when we use a TCP socket inside a container to proxy to Docker. powerstrip v0.0.2 instead listens on a unix socket. However, this means that to use Swarm - socat or an equivalent proxy needs to be used (to forward TCP:2375 -> UNIX:/var/run/docker.sock

The candidate for v0.0.2 that uses the unix socket is on DockerHub with the unix-socket tag.

$ docker pull clusterhq/powerstrip:unix-socket

And then:

$ docker run --rm --name powerstrip \
    -v /var/run:/host-var-run \
    -v /etc/powerstrip-demo/adapters.yml:/etc/powerstrip/adapters.yml \
    --link powerstrip-flocker:flocker \
    --link powerstrip-weave:weave \
    clusterhq/powerstrip:unix-socket

The additional change required is to tell docker to listen on the /var/run/docker.real.sock

This can be done with the arguments to the docker daemon docker -D -H unix:///var/run/docker.real.sock

[aanm]

@binocarlos It's odd you're saying that because I'm doing a docker-compose up (which I believe has a attach and it works flawless with clusterhq/powerstrip:latest

[binocarlos]

Hmm - perhaps it is something else - the issue was with commands like this:

$ docker run ubuntu echo hello

And the output would be nothing (because the attach step to receive the text "hello" was breaking)... But when stdin is attached - it seemed to work:

$ docker run -i ubuntu echo hello

Perhaps compose is attaching stdin?

There are some commits between :v0.0.1 and :latest (master) - perhaps this is why it is working for :latest and not :v0.0.1?

[aanm]

@binocarlos The only thing I know is that compose up does this:

By default, `docker-compose up` will aggregate the output of each container, and
when it exits, all containers will be stopped. If you run `docker-compose up -d`,
it'll start the containers in the background and leave them running.