travis-ci / cyclist

AWS ASG lifecycle thing :tada: :bicyclist:
MIT License
8 stars 2 forks source link

Instance implosion terminations result in Terminating:Wait #30

Closed meatballhat closed 7 years ago

meatballhat commented 7 years ago

When an instance "implodes", meaning that it performs a halt based on a problem that it detects itself such as the occurrence of unregister_netdevice errors, the stop hook request to /terminations/{instance_id} responds generically with an error due to the absence of an AWS-triggered termination lifecycle action. Once the halted instance is deemed "Unhealthy", then an AWS-triggered termination lifecycle action is created, yet there is no instance remaining to confirm the termination.

In such cases, cyclist should handle the instance "implosion" such that when the termination SNS event comes in that it may be immediately confirmed so that the instance state does not transition to Terminating:Wait until the standard timeout period elapses.