search

mapbox / magic-cfn-resources

Build Lambda-backed custom CloudFormation resources ✨
4 stars 5 forks source link

Issues with spot fleet resource during update rollback #21

Open rclark opened 7 years ago

rclark commented 7 years ago

I encountered a scenario last night where a stack update attempted to change the fleet's Launch Specification. However there were syntax errors, and so the update to the custom resource failed. The result was that when the update rollback was complete, the new fleet was considered obsolete and the old fleet request was canceled. This led our system to terminate EC2s in the new spot fleet as soon as they launched.

The steps that led to this were complex, but after conversation with AWS support, the takeaways are:

  1. When the custom resource's lambda function receives an UPDATE request, it must compare the incoming set of properties to the existing spot fleet. If they are identical, then it should not create a new spot fleet, and should not provide CloudFormation with a new spot fleet request ID.

  2. "Ref" to custom cfn resources is actually not supported or documented. You're supposed to use "Fn::GetAtt:" for any and all information that you need to pull from the custom resource. The fact that we use a "Ref" to the spot fleet resource as a way to derive the existing fleet's request ID is "a loophole" and it can't be trusted.

cc @mapbox/platform

jakepruitt commented 7 years ago

😱

emilymcafee commented 7 years ago

bitmoji