From time to time, there are docker nodes that remain in the docker swarm forever:
docker node in Drain mode, without any label
Current theory is something along the line of:
a user starts a service,
autoscaling creates an EC2,
the EC2 connects to the swarm,
the user deletes the service,
autoscaling terminates the EC2 without setting the node labels,
or:
some EC2 that took too long to connect.
--> the docker node remain visible in the swarm forever and autoscaling will never remove it since it lacks the node labels
docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
s1aics6zjb5mp1r1r3fqn9dv2 ip-10-1-6-138 Down Drain 25.0.3
o6ny1i7978a6ybusp7mti9os7 ip-10-1-7-47 Ready Active 25.0.3
0yq3i0z3wx3fa9y2vducgt937 * prod-man1 Ready Active Leader 25.0.3
lgpuh0fylqrxdrarvhxcsd14w prod-ops1 Ready Active 25.0.3
kx1cn9gw2blfdct2x34h6ryr4 prod-sim1 Ready Active 25.0.3
From time to time, there are docker nodes that remain in the docker swarm forever:
Current theory is something along the line of:
or: some EC2 that took too long to connect.
--> the docker node remain visible in the swarm forever and autoscaling will never remove it since it lacks the node labels