If a node is in "drain" mode, tasks can be orphaned, and are not cleaned up. I noticed this when using docker stack deploy, but possibly there's other ways to arrive in this situation
Steps to reproduce the issue:
On a single-node swarm, put the node in availability drain:
docker node update --availability=drain $(docker node ls --quiet)
docker stack ps mystack
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
l642i9p5gs4c mystack_web.1 nginx:alpine Running Pending less than a second ago "no suitable node (1 node not …"
0vqk2i4xdgeg pnsu2xb5xavnw5q7vmabygz8y.1 nginx:alpine Remove Pending 5 minutes ago "no suitable node (1 node not …"
Notice that one more randomly named task is added:
docker stack ps mystack
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
v99e6svzbw1l mystack_web.1 nginx:alpine Running Pending less than a second ago "no suitable node (1 node not …"
l642i9p5gs4c ma35f4bhbygnw7exbrkqw3oct.1 nginx:alpine Remove Pending about a minute ago "no suitable node (1 node not …"
0vqk2i4xdgeg pnsu2xb5xavnw5q7vmabygz8y.1 nginx:alpine Remove Pending 7 minutes ago "no suitable node (1 node not …"
Describe the results you received:
Each re-deploy adds a new task with desired state Remove. These tasks are never removed as long as the node is in "drain" mode, even if the service is removed.
Note that no containers are seen for those tasks
Updating the node to active removes the old tasks
docker service ps does not show these tasks;
$ docker service ps mystack_web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
hhzfa05q7kqf mystack_web.1 nginx:alpine Running Pending 6 minutes ago "no suitable node (1 node not …"
Description
If a node is in "drain" mode, tasks can be orphaned, and are not cleaned up. I noticed this when using
docker stack deploy
, but possibly there's other ways to arrive in this situationSteps to reproduce the issue:
On a single-node swarm, put the node in availability
drain
:Deploy a stack:
Watch the output of
docker stack ps mystack
:Remove the stack:
And re-deploy the stack
Notice that one more randomly named task is added:
Describe the results you received:
Each re-deploy adds a new task with desired state
Remove
. These tasks are never removed as long as the node is in "drain" mode, even if the service is removed.active
removes the old tasksdocker service ps
does not show these tasks;Each task also belongs to a different service;
Describe the results you expected:
Tasks to be removed
Additional information you deem important (e.g. issue happens only occasionally):
Output of
docker version
:Output of
docker info
:Additional environment details (AWS, VirtualBox, physical, etc.):
Docker for Mac