djeeg
commented
6 years ago I thought I was fine just ignoring the extra container, considering it was from a stack that had been removed and re-deployed
Until connecting to the redis service started to fail in some nodejs services
Error: connect ECONNREFUSED 10.0.29.14:7000
I checked the state to see if the ghost container is still there, yep it is, still alive after 8 days.
$ docker ps | grep "redis"
9e293525f7d4 "docker-entrypoint.s…" 30 hours ago Up 30 hours (healthy) 6379/tcp, 7000/tcp redis.1.ugjl5t7494a1fly5c0emodyz9
eff366c1acb0 "docker-entrypoint.s…" 8 days ago Up 8 days (healthy) 6379/tcp, 7000/tcp redis.1.pfma6polo81rpxx9cxoz4pqt5It's odd, because the container is still hanging around in the stack, though it is now marked as Remove and the ID has changed to a hash code
$ docker stack ps --no-trunc data
ugjl5t7494a1fly5c0emodyz9 redis.1 swarm-manager000000 Running Running 30 hours ago
pfma6polo81rpxx9cxoz4pqt5 ydqo5bc<removed>.1 swarm-manager000000 Remove Running 8 days agoStack replicas look okay
$ docker stack services data
ID NAME MODE REPLICAS
r22m0o2v5meo redis replicated 1/1 To debug why the nodejs services were failing, I thought to jump onto the container and check the network
$ docker exec -it 9e293525f7d4 /bin/bash
root@9e293525f7d4:/data# getent hosts
127.0.0.1 localhost
127.0.0.1 localhost ip6-localhost ip6-loopback
10.0.29.36 9e293525f7d4Hang on, this IP is different from the IP reported in the nodejs error
I'll try a different node service, dump a few container ids to test
Welcome to Docker!
swarm-worker000000:~$ docker ps | grep microse
4c6c48729de0 "/tini -- /start.sh" 30 minutes ago Up 30 minutes microservices1_2_serviceauths.1.qggswpvudcqlb6kgmb5yyk0rp
f1a114ceaa23 "/tini -- /start.sh" 30 minutes ago Up 30 minutes microservices1_2_serviceapi.1.82x3tzwadm43sk6fdubdoph3a swarm-worker000000:~$ docker exec -it $(docker ps | grep "_serviceapi.1" | awk '{print $1}') /bin/bash
root@f1a114ceaa23:/serviceroot# ping redis
PING redis (10.0.29.14): 56 data bytesswarm-worker000000:~$ docker exec -it $(docker ps | grep "_serviceauths.1" | awk '{print $1}') /bin/bash
root@4c6c48729de0:/serviceroot# ping redis
PING redis (10.0.29.35): 56 data bytesOkay the nodejs error is making sense, there is an extra IP in the DNS, I am guessing the ghost/orphaned container
$ getent hosts tasks.redis
10.0.29.15 tasks.redis
10.0.29.36 tasks.redisTry to connect to the orphaned container
$ docker exec -it eff366c1acb0 /bin/bash
OCI runtime exec failed: exec failed: container_linux.go:296: starting container process caused "process_linux.go:86: executing setns process caused \"exit status 21\"": unknownNot sure how to force remove this container from the service
FrenchBen
Hi,
Yesterday I went ahead and performed a full swarm rebuild of my azure swarm The migration from 17.09-ish to 17.12.0 seemed to go smoothly I got everything working correctly before knocking off of the day
This morning I found the application isnt working and see lots of errors from a redis service
First I try restarting the service
However the service doesnt scale down
I can still see the container running
I try removing the stack completely, however the container is still running
Next I try redeploying the stack
Seems to bring the application back to life, though i now have another ghost container
Try to end the container, command hangs the console
$ docker kill eff366c1acb0Do some digging into docker logs, these seems related to the redis error [could not fetch metadata: cannot read metadata: open /mnt/cloudstor/cloudstor-metadata] [no such file or directory]
I see that [Failed to deserialize netlink ndmsg: Link not found] has been fixed with https://github.com/docker/libnetwork/issues/2045, could it be related to the cloudstor failure?
Current stack config (though I will change this now to replace cloudstor with a local volume)
And redis config
docker info