ITISFoundation / osparc-simcore

🐼 osparc-simcore simulation framework
https://osparc.io
MIT License
46 stars 27 forks source link

Autoscaling: Sometimes unlabelled docker nodes remain in the swarm forever #6141

Open sanderegg opened 3 months ago

sanderegg commented 3 months ago

From time to time, there are docker nodes that remain in the docker swarm forever:

Current theory is something along the line of:

or: some EC2 that took too long to connect.

--> the docker node remain visible in the swarm forever and autoscaling will never remove it since it lacks the node labels

docker node ls
ID                            HOSTNAME        STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
s1aics6zjb5mp1r1r3fqn9dv2     ip-10-1-6-138   Down      Drain                           25.0.3
o6ny1i7978a6ybusp7mti9os7     ip-10-1-7-47    Ready     Active                          25.0.3
0yq3i0z3wx3fa9y2vducgt937 *   prod-man1       Ready     Active         Leader           25.0.3
lgpuh0fylqrxdrarvhxcsd14w     prod-ops1       Ready     Active                          25.0.3
kx1cn9gw2blfdct2x34h6ryr4     prod-sim1       Ready     Active                          25.0.3

Image

sanderegg commented 2 months ago

I found out that this happens when:

for info @YuryHrytsuk