can't kill or restart ghosted containers (returns status 255)

[dweinstein]

This started happening when I restarted the VM I believe.

$ docker ps
CONTAINER ID        IMAGE                           COMMAND                CREATED             STATUS              PORTS                    NAMES
128cb396bf69        coreos/etcd:latest              /opt/etcd/bin/etcd     26 hours ago        Ghost               4001/tcp, 7001/tcp       service_etcd           

I tried killing all ghosted containers:

docker@boot2docker:/var/lib/docker$ docker ps | grep Ghost | awk '{print $1}' | xargs docker kill

Which resulted in errors like the following:

$ docker kill 128cb396bf69
Error: kill: Cannot kill container 128cb396bf69: exit status 255
2014/01/25 21:22:11 Error: failed to kill one or more containers

Things I tried:

After ssh'ing into the dvm (with dvm ssh) I ended up noticing the date was way off, so I adjusted it to match my host system. I also thought maybe there was an aufs mount issue so I did a docker@boot2docker:~$ mount | awk '{print $3}' | grep aufs | xargs sudo umount (don't know if this is recommended...)

I manually edited /usr/local/etc/init.d/docker to pass the -D option to docker when it starts (I guess this is also configurable in ${HOME}/.dvm/dvm.conf but I liked my way at the time.

I looked at the log file in /var/lib/docker/docker.log and saw the following:

[debug] api.go:1038 Calling POST /containers/{name:.*}/kill
2014/01/26 03:02:57 POST /v1.8/containers/128cb396bf69/kill
[/var/lib/docker|31a8991b] +job kill(128cb396bf69)
2014/01/26 03:02:57 error killing container 128cb396bf69 (lxc-kill: failed to get the init pid, exit status 255)
Cannot kill container 128cb396bf69: exit status 255[/var/lib/docker|31a8991b] -job kill(128cb396bf69) = ERR (1)
[error] api.go:1064 Error: kill: Cannot kill container 128cb396bf69: exit status 255
[error] api.go:87 HTTP Error: statusCode=500 kill: Cannot kill container 128cb396bf69: exit status 255

Anything else I should try?

[dweinstein]

I ended up doing a dvm destroy and starting over. If anyone does figure this out let me know.

[fnichol]

It looks like this issue is addressed with steeve/boot2docker#17 which is incorporated in release 0.4.0/0.4.1 of dvm (includes boot2docker 0.5.4). I'm going to (fingers crossed) close this but we can re-open if the problem persists. Better still, if it does persist it's something we may need to address in boot2docker. Thank you!