search

docker-archive / for-aws

92 stars 26 forks source link

Changing worker instance type via Cloud Formation stack template update causes downtime #165

Open mrmachine opened 6 years ago

mrmachine commented 6 years ago

Expected behavior

No downtime as a new node is started first, then the old node is drained and its services are rescheduled to the new node, for each worker in the pool.

Actual behavior

Old nodes are terminated before services (with health checks) are running/healthy on new nodes.

Information

Docker for AWS 18.03.0-ce

Steps to reproduce the behavior

  1. Create a Docker for AWS stack with the default settings.
  2. Deploy some services to it.
  3. Update the stack and change worker instance type or worker storage.
  4. Watch all your sites go down for several minutes after the nodes are terminated, while swarm is still trying to bring services back up.