Docker ps filtering is not correct

[gbmeuk]

I have found that there is a divergence in the state response returned from sdc-docker when compared to standard docker engine. This means that when runningdocker-compose scale correct checks against ''stopped=False'' in the docker-compose code is failing to correctly identify the number of running containers. For example

Running against sdc-docker

(gaz@gMac)-(14:15:22)-(dockerlx)
$docker info
Containers: 13
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1000
Storage Driver: sdc
 SDCAccount: gaz
Logging Driver: json-file
Plugins: 
 Volume: 
 Network: 
Swarm: 
 NodeID: 
 Is Manager: false
 Node Address: 
Init Binary: 
containerd version: 
runc version: 
init version: 
Kernel Version: 3.12.0-1-amd64
Operating System: SmartDataCenter
OSType: linux
Architecture: x86_64
CPUs: 0
Total Memory: 0 B
Name: GB-Home
ID: 3bc1d32f-43da-4afe-b8a9-88726bf3d3c2
Docker Root Dir: 
Debug Mode (client): false
Debug Mode (server): false
Username: gbmeuk
Registry: https://index.docker.io/v1/
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Live Restore Enabled: false
(gaz@gMac)-(14:15:29)-(dockerlx)
$
(gaz@gMac)-(12:25:55)-(dockerlx)
$docker-compose start
Starting consul ... 
Starting consul ... done
(gaz@gMac)-(13:06:42)-(dockerlx)
$docker-compose ps

               Name                               Command                               State                                Ports                
-------------------------------------------------------------------------------------------------------------------------------------------------
dockerlx_consul_1                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_2                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_3                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
(gaz@gMac)-(13:07:30)-(dockerlx)
$
(gaz@gMac)-(13:07:31)-(dockerlx)
$
(gaz@gMac)-(13:07:31)-(dockerlx)
$docker-compose scale consul=3

Desired container number already achieved
(gaz@gMac)-(13:07:38)-(dockerlx)
$
(gaz@gMac)-(13:13:04)-(dockerlx)
$
(gaz@gMac)-(13:13:04)-(dockerlx)
$
(gaz@gMac)-(13:13:05)-(dockerlx)
$docker-compose scale consul=7

Creating and starting dockerlx_consul_4 ... 
Creating and starting dockerlx_consul_5 ... 
Creating and starting dockerlx_consul_6 ... 
Creating and starting dockerlx_consul_7 ... 

Creating and starting dockerlx_consul_4 ... done
Creating and starting dockerlx_consul_5 ... done
Creating and starting dockerlx_consul_6 ... done
Creating and starting dockerlx_consul_7 ... done
(gaz@gMac)-(13:15:05)-(dockerlx)
$
(gaz@gMac)-(13:15:05)-(dockerlx)
$

(gaz@gMac)-(13:18:24)-(dockerlx)
$docker stop dockerlx_consul_1 dockerlx_consul_2 dockerlx_consul_3 dockerlx_consul_4
dockerlx_consul_1
dockerlx_consul_2
dockerlx_consul_3
dockerlx_consul_4
(gaz@gMac)-(13:19:12)-(dockerlx)
$docker-compose ps
               Name                               Command                               State                                Ports                
-------------------------------------------------------------------------------------------------------------------------------------------------
dockerlx_consul_1                    /usr/local/bin/containerpi ...       Exit -9                              53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_2                    /usr/local/bin/containerpi ...       Exit -9                              53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_3                    /usr/local/bin/containerpi ...       Exit -9                              53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_4                    /usr/local/bin/containerpi ...       Exit -9                              53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_5                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_6                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_7                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
(gaz@gMac)-(13:19:33)-(dockerlx)
$docker-compose scale consul=3
Stopping and removing dockerlx_consul_4 ... 
Stopping and removing dockerlx_consul_5 ... 
Stopping and removing dockerlx_consul_6 ... 
Stopping and removing dockerlx_consul_7 ... 
Stopping and removing dockerlx_consul_4 ... done
Stopping and removing dockerlx_consul_5 ... done
Stopping and removing dockerlx_consul_6 ... done
Stopping and removing dockerlx_consul_7 ... done
(gaz@gMac)-(13:21:39)-(dockerlx)
$docker-compose ps

               Name                               Command                               State                                Ports                
-------------------------------------------------------------------------------------------------------------------------------------------------
dockerlx_consul_1                    /usr/local/bin/containerpi ...       Exit -9                              53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_2                    /usr/local/bin/containerpi ...       Exit -9                              53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_3                    /usr/local/bin/containerpi ...       Exit -9                              53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp

Whereas on docker-engine, this doesn't happen

(gaz@gMac)-(14:15:13)-(dockerlx)
$docker info
Containers: 14
 Running: 4
 Paused: 0
 Stopped: 10
Images: 15
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 41
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: 
containerd version: 
runc version: 
init version: 
Security Options:
 seccomp
Kernel Version: 4.4.16-boot2docker
Operating System: Boot2Docker 1.12.0 (TCL 7.2); HEAD : e030bab - Fri Jul 29 00:29:14 UTC 2016
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.955 GiB
Name: default
ID: CJ4P:LB4U:DUNM:L254:3QQ2:4V6K:P4JD:QU64:36CW:OVAS:3EQM:5APR
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 34
 Goroutines: 60
 System Time: 2017-02-01T14:15:59.7468881Z
 EventsListeners: 0
Username: gbmeuk
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
(gaz@gMac)-(13:35:35)-(dockerlx)
$docker-compose up -d
Creating dockerlx_consul_1
(gaz@gMac)-(13:35:53)-(dockerlx)
$
(gaz@gMac)-(13:39:09)-(dockerlx)
$
(gaz@gMac)-(13:39:09)-(dockerlx)
$docker-compose scale consul=7
Creating and starting dockerlx_consul_2 ... done
Creating and starting dockerlx_consul_3 ... done
Creating and starting dockerlx_consul_4 ... done
Creating and starting dockerlx_consul_5 ... done
Creating and starting dockerlx_consul_6 ... done
Creating and starting dockerlx_consul_7 ... done
(gaz@gMac)-(13:39:24)-(dockerlx)
$docker-compose ps
               Name                               Command                               State                                Ports                
-------------------------------------------------------------------------------------------------------------------------------------------------
dockerlx_consul_1                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:32772->8500/tcp            
dockerlx_consul_2                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:32773->8500/tcp            
dockerlx_consul_3                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:32777->8500/tcp            
dockerlx_consul_4                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:32776->8500/tcp            
dockerlx_consul_5                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:32778->8500/tcp            
dockerlx_consul_6                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:32774->8500/tcp            
dockerlx_consul_7                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:32775->8500/tcp            
(gaz@gMac)-(13:39:45)-(dockerlx)
$docker stop dockerlx_consul_1 dockerlx_consul_2 dockerlx_consul_3 dockerlx_consul_4
dockerlx_consul_1
dockerlx_consul_2
dockerlx_consul_3
dockerlx_consul_4
(gaz@gMac)-(13:40:05)-(dockerlx)
$
(gaz@gMac)-(13:40:07)-(dockerlx)
$
(gaz@gMac)-(13:40:07)-(dockerlx)
$docker-compose ps
               Name                               Command                               State                                Ports                
-------------------------------------------------------------------------------------------------------------------------------------------------
dockerlx_consul_1                    /usr/local/bin/containerpi ...       Exit 137                                                                
dockerlx_consul_2                    /usr/local/bin/containerpi ...       Exit 137                                                                
dockerlx_consul_3                    /usr/local/bin/containerpi ...       Exit 137                                                                
dockerlx_consul_4                    /usr/local/bin/containerpi ...       Exit 137                                                                
dockerlx_consul_5                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:32778->8500/tcp            
dockerlx_consul_6                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:32774->8500/tcp            
dockerlx_consul_7                    /usr/local/bin/containerpi ...       Up                                   53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:32775->8500/tcp            
(gaz@gMac)-(13:40:09)-(dockerlx)
$docker-compose scale consul=3
Desired container number already achieved
(gaz@gMac)-(13:40:20)-(dockerlx)

We've noticed that there is a difference in the State object returned when inspecting the container.

On sdc-docker:

(gaz@gMac)-(13:50:18)-(dockerlx)
$docker inspect dockerlx_consul_1 | json -a State
{
  "Error": "",
  "ExitCode": -9,
  "FinishedAt": "2017-02-01T13:18:59.451Z",
  "OOMKilled": false,
  "Paused": false,
  "Pid": 0,
  "Restarting": false,
  "Running": false,
  "StartedAt": "0001-01-01T00:00:00Z"
}

And on docker-engine:

(gaz@gMac)-(14:15:07)-(dockerlx)
$docker inspect dockerlx_consul_1 | json -a State
{
  "Status": "exited",
  "Running": false,
  "Paused": false,
  "Restarting": false,
  "OOMKilled": false,
  "Dead": false,
  "Pid": 0,
  "ExitCode": 137,
  "Error": "",
  "StartedAt": "2017-02-01T13:35:53.272824591Z",
  "FinishedAt": "2017-02-01T13:40:05.782494833Z"
}

We expect there is some mismatch with what is considered the current state with the Docker container state class

[tgross]

A more minimal repro seems to point to the filter for the stopped status.

$ docker run -d --name test alpine:3.5 sleep 86400
$ docker ps --filter="status=running" # verify we're provisioned and running first
CONTAINER ID        IMAGE             COMMAND             CREATED             STATUS         PORTS   NAMES
45bfd5b330b4        alpine:3.5        "sleep 86400"       23 seconds ago      Up 9 seconds           test

$ docker stop test
test
$ docker ps --filter="status=stopped"
Error response from daemon: Unrecognised filter value for status (045f6ffe-eb15-4b69-b580-dfbd71ccbf97)

All the other filters in https://github.com/docker/docker/commit/ea09f036824aee74112a75ac99579ae4f10251b5 appear to be working correctly.

[gbmeuk]

submitted pull request #111

[twhiteman]

@tgross

$ docker ps --filter="status=stopped" Error response from daemon: Unrecognised filter value for status (045f6ffe-eb15-4b69-b580-dfbd71ccbf97)

That is not correct, as docker/docker also does the same thing with the filter status value:

docker ps --filter="status=stopped"
Error response from daemon: Unrecognised filter value for status: stopped

Docker's server currently accepts these status values: https://github.com/docker/docker/blob/b59ee9486fad5fa19f3d0af0eb6c5ce100eae0fc/container/state.go#L150

so by that it looks like sdc-docker is missing dead and removing status fields... and to be honest I'm not sure if adding these will solve the original reported problem.

[gbmeuk]

removed previous comment about #111 working. Tests weren't thorough enough.

Only obvious thing that still stands out for me is the missing "Status" and "Dead" keys in the sdc-docker State object returned from docker inspect. Also, the "StartedAt" and "FinishedAt" values alternately have "0001-01-01T00:00:00Z" as their value depending on whether the container is stopped or running. This differs from docker engine

[gbmeuk]

So following some of the logs, this seems to be an issue of not correctly handling the docker -a flag when passing filters. For example:

(gaz@gMac)-(13:30:30)-(dockerlx)
$docker ps --all=true -n=-1 --no-trunc=false --size=false --filter label=com.docker.compose.project=dockerlx -f label=com.docker.compose.service=consul -f label=com.docker.compose.oneoff=False
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS                                                                            NAMES
2a55b51de079        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_2
e61cf429d85c        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_3
245a8582694e        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_1
(gaz@gMac)-(13:30:34)-(dockerlx)
$docker ps --all=false -n=-1 --no-trunc=false --size=false --filter label=com.docker.compose.project=dockerlx -f label=com.docker.compose.service=consul -f label=com.docker.compose.oneoff=False
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS                                                                            NAMES
2a55b51de079        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_2
e61cf429d85c        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_3
245a8582694e        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_1
(gaz@gMac)-(13:30:42)-(dockerlx)
$

whereas when running the same commands on docker-engine:

(gaz@gMac)-(11:47:11)-(dockerlx)
$docker ps --all=false -n=-1 --no-trunc=false --size=false --filter label=com.docker.compose.project=dockerlx -f label=com.docker.compose.service=consul -f label=com.docker.compose.oneoff=False
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
(gaz@gMac)-(11:47:12)-(dockerlx)
$docker ps --all=true -n=-1 --no-trunc=false --size=false --filter label=com.docker.compose.project=dockerlx -f label=com.docker.compose.service=consul -f label=com.docker.compose.oneoff=False
CONTAINER ID        IMAGE                    COMMAND                  CREATED              STATUS                            PORTS               NAMES
a1bcaaa1de6f        mesoform/consul:latest   "/usr/local/bin/co..."   About a minute ago   Exited (137) About a minute ago                       dockerlx_consul_3
d55839b4ca86        mesoform/consul:latest   "/usr/local/bin/co..."   About a minute ago   Exited (137) About a minute ago                       dockerlx_consul_2
260a5df0c03a        mesoform/consul:latest   "/usr/local/bin/co..."   About a minute ago   Exited (137) About a minute ago                       dockerlx_consul_1
(gaz@gMac)-(11:47:19)-(dockerlx)
$

And without filters:

(gaz@gMac)-(13:36:46)-(dockerlx)
$docker ps --all=false -n=-1 --no-trunc=false --size=false | grep consul
(gaz@gMac)-(13:37:22)-(dockerlx)
$docker ps --all=true -n=-1 --no-trunc=false --size=false | grep consul
2a55b51de079        mesoform/consul                                    "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_2
e61cf429d85c        mesoform/consul                                    "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_3
245a8582694e        mesoform/consul                                    "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_1
(gaz@gMac)-(13:37:32)-(dockerlx)
$

This is the same with any filter:

(gaz@gMac)-(13:38:18)-(dockerlx)
$docker ps --all=false -n=-1 --no-trunc=false --size=false --filter name=dockerlx_consul_*
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS                                                                            NAMES
2a55b51de079        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_2
e61cf429d85c        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_3
245a8582694e        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_1
(gaz@gMac)-(13:38:45)-(dockerlx)
$

except of course "exited":

(gaz@gMac)-(13:39:30)-(dockerlx)
$docker ps --all=false -n=-1 --no-trunc=false --size=false --filter label=com.docker.compose.project=dockerlx -f label=com.docker.compose.service=consul -f label=com.docker.compose.oneoff=False --filter status=running
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
(gaz@gMac)-(13:39:41)-(dockerlx)
$docker ps --all=false -n=-1 --no-trunc=false --size=false --filter label=com.docker.compose.project=dockerlx -f label=com.docker.compose.service=consul -f label=com.docker.compose.oneoff=False --filter status=exited
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                     PORTS                                                                            NAMES
2a55b51de079        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_2
e61cf429d85c        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_3
245a8582694e        mesoform/consul     "/usr/local/bin/co..."   24 hours ago        Exited (-9) 12 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_1
(gaz@gMac)-(13:39:48)-(dockerlx)
$

[gbmeuk]

I've tried playing around with some of the logic between getContainers() and listDockerVms() and making certain changes around the usage of opts.all has a clear effect but I'm not seeing where is has any relationship with filters. I also don't have a javascript plugin for my IDE to help understand what is going on better. Maybe someone else can shed some light?

[twhiteman]

@gbmeuk Nice digging - it is indeed the label filters where I think the sdc-docker code is causing a problem.

From the docker API docs: https://docs.docker.com/engine/api/v1.23/#containers some query parameters passed to docker ps API result in a check against all containers, whilst others are only against running containers, for example docker ps --limit 10 will return stopped containers (even though you did not pass the -a flag).

Now here's the rub, it seems that filter is special, it returns a match against all containers if you specify the status filter (docker ps --filter status=X), but not when you specify another filter like label (docker ps --filter label=Y). This is where sdc-docker gets it wrong, as even for label filters, it is returning against all matches.

The code that handles the specifics of this are here: https://github.com/joyent/sdc-docker/blob/master/lib/endpoints/containers.js#L89

I can write up a patch that should resolve this - ensuring the all flag is only set for status filters and not on all filters.

[twhiteman]

Should be solved with the above change.

[gbmeuk]

Something is still differing. Running:

docker-compose ps

(gaz@gMac)-(12:54:52)-(dockerlx)
$docker-compose ps
  SubjectAltNameWarning
               Name                               Command                               State                                Ports                
-------------------------------------------------------------------------------------------------------------------------------------------------
dockerlx_consul_1                    /usr/local/bin/containerpi ...       Exit 0                               53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_2                    /usr/local/bin/containerpi ...       Exit 1                               53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
dockerlx_consul_3                    /usr/local/bin/containerpi ...       Exit 1                               53/tcp, 53/udp, 8300/tcp,          
                                                                                                               8301/tcp, 8301/udp, 8302/tcp,      
                                                                                                               8302/udp, 8400/tcp,                
                                                                                                               0.0.0.0:8500->8500/tcp             
(gaz@gMac)-(12:55:40)-(dockerlx)
$

performs the following API request (shown in sdc-docker logs) GET /v1.22/containers/json?all=0&limit=-1&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Ddockerlx%22%2C+%22com.docker.compose.oneoff%3DTrue%22%5D%7D&trunc_cmd=0&size=0 HTTP/1.1

As does docker-compose scale (runs with all=0)

(gaz@gMac)-(12:55:40)-(dockerlx)
$docker-compose scale consul=3
Desired container number already achieved
(gaz@gMac)-(12:57:44)-(dockerlx)
$

logs GET /v1.22/containers/json?all=0&limit=-1&filters=%7B%22label%22%3A+%5B%22com.docker.compose.project%3Ddockerlx%22%2C+%22com.docker.compose.service%3Dconsul%22%2C+%22com.docker.compose.oneoff%3DFalse%22%5D%7D&trunc_cmd=0&size=0 HTTP/1.1

Whereas docker ps --all=false ... (with patch)

(gaz@gMac)-(13:00:22)-(dockerlx)
$docker ps --all=false -n=-1 --no-trunc=false --size=false --filter label=com.docker.compose.project=dockerlx -f label=com.docker.compose.service=consul -f label=com.docker.compose.oneoff=False
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
(gaz@gMac)-(13:00:51)-(dockerlx)
$

calls the API as follows GET /v1.24/containers/json?filters=%7B%22label%22%3A%7B%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Ddockerlx%22%3Atrue%2C%22com.docker.compose.service%3Dconsul%22%3Atrue%7D%7D HTTP/1.1

and with --all=true

$docker ps --all=true -n=-1 --no-trunc=false --size=false --filter label=com.docker.compose.project=dockerlx -f label=com.docker.compose.service=consul -f label=com.docker.compose.oneoff=False
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                    PORTS                                                                            NAMES
baac29e3c4cc        mesoform/consul     "/usr/local/bin/co..."   2 days ago          Exited (1) 25 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_3
605b2a1884de        mesoform/consul     "/usr/local/bin/co..."   2 days ago          Exited (1) 25 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_2
c194b140e193        mesoform/consul     "/usr/local/bin/co..."   2 days ago          Exited (0) 25 hours ago   53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   dockerlx_consul_1
(gaz@gMac)-(13:03:31)-(dockerlx)
$

calls GET /v1.24/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Ddockerlx%22%3Atrue%2C%22com.docker.compose.service%3Dconsul%22%3Atrue%7D%7D HTTP/1.1

[gbmeuk]

@twhiteman @tgross @joshwilsdon, I'm afraid I don't seem to have the ability to reopen this issue. It seems like what was fixed was a related but different bug.

[twhiteman]

@gbmeuk Indeed, my bad - docker-compose scale still seems to be counting stopped containers.

[twhiteman]

Docker compose is passing limit=-1, which is thus enabling the all flag in sdc-docker.

[gbmeuk]

Thanks. So how/why does limit=-1 enable the all flag?

[twhiteman]

@gbmeuk The docker API docs just say using limit enables all containers: https://docs.docker.com/engine/api/v1.23/#containers

and sdc-docker takes that literally, so if a limit (any limit) is used, it would filter against all containers: https://github.com/joyent/sdc-docker/blob/master/lib/endpoints/containers.js#L89

[twhiteman]

FYI, a sdc-docker change is under review here: https://cr.joyent.us/#/c/1441/

[trentm]

I am defeated by docker docs (useless for giving intent of options to /containers/json and/or docker ps). The core logic seems to be in docker.git:daemon/list.go#foldFilter.

[twhiteman]

Closing, again... at least I tested it with docker-compose this time ;)

[gbmeuk]

I can see it was understood that way. Validated working on my environment as well. Thanks.