Closed MikeTheCanuck closed 6 years ago
It is possible that Ian has somehow massaged the unhealthy EC2 host sufficiently to be able to accept new containers - there are currently a handful of containers assigned to that host, some of which have survived the initial HealthCheck timeout period.
Attempted a number of deploys of the
master.yaml
using a change set in CloudFormation over the past couple of days. They have failed, but failed in a weird way that we haven't observed before, and the UPDATE_ROLLBACK takes forever to complete (as opposed to typical failures, which rollback in under ten minutes).The following type of error is observed for three of the services in our stack - 2018HA, 2018LE, 2018ND (housing-affordability, local-elections and neighborhood-development):
The prevailing theory right now is that the update is failing because the deploy of current task definitions isn't succeeding, because one of the two EC2 hosts for our containers is not healthy, and is not able to launch containers.
Consequences