search

moby / swarmkit

A toolkit for orchestrating distributed systems at any scale. It includes primitives for node discovery, raft-based consensus, task scheduling and more.
Apache License 2.0
3.34k stars 612 forks source link

How to kill your manager #1628

Open FrenchBen opened 7 years ago

FrenchBen commented 7 years ago

Deploy Docker for AWS on t2.micro instances, with the following setup:

Then create 1 simple nginx service with 1000 replicas:

docker service create --name kite --replicas 1000 -p 80:80 kitematic/hello-world-nginx

Observe your resources being killed.

cat

dperny commented 7 years ago

Can reproduce. My manager became kill. Not just docker, the whole node was kaput; I couldn't ssh in to even check its status

@thaJeztah @crosbymichael

mrjana commented 7 years ago

@dperny Can you go a little low on the number of replicas to may be at least get the node into some bad state but not all that bad that even basic debugging becomes harder?

abronan commented 7 years ago

@dperny @FrenchBen

FWIW, a bunch of related issues/comments in #1156, docker/docker#24027 and docker/docker#24651

A reasonable thing to do to circumvent this issue is to opt-out the Manager as being able to host tasks with --availability drain. See the admin guide.