Unexpected Cloudformation-generated physical resource ID on stack rollback of in-flight custom resources

aws-cloudformation / cloudformation-coverage-roadmap

The AWS CloudFormation Public Coverage Roadmap

Creative Commons Attribution Share Alike 4.0 International

1.11k stars 56 forks source link

Name of the resource

AWS::CloudFormation::CustomResource

Resource name

No response

Reference Link

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/crpg-ref-requesttypes-delete.html

Details

I am observing behavior in Custom Resources that I have been unable to find anywhere else on either the web or AWS documentation. It involves the presence of a value in PhysicalResourceId that I suspect CloudFormation is generating on-the-fly under specific rollback scenarios.

This behavior only surfaces during the rollback of a stack creation operation in which a custom resource began to be provisioned prior to the rollback-inducing event.

Here's the sequence of events from CloudFormation's perspective:

CFN begins creating resources, one of which is an AWS::CloudFormation::CustomResource
While the custom resource is still CREATE_IN_PROGRESS, another resource fails to be created
CFN begins to roll back the stack creation
CFN emits a CREATE_FAILED event (with reason Resource creation cancelled) for the custom resource
CFN begins deleting the custom resource (event: DELETE_IN_PROGRESS)

On the custom resource side, I've observed the following:

I get a Delete request for the custom resource (makes sense)
The Delete request is sent before the Create request response was ever received by CFN (makes less sense)
The Delete request contains a physical resource ID that I never set on the resource. It's always a concatenation of a) the stack name, b) the logical ID, and c) some random hash. For example: MYSTACK-LOGICALID-B36FTLCJN97H

I've been unable to find any documentation explaining the pseudo-physical resource ID that CloudFormation includes on Delete requests sent due to a stack rollback (if that is indeed the intended scenario where this occurs). All documentation implies that the PhysicalResourceId field is always a value that was provided by the custom resource provider after a successful Create request.

I would like to bump this as I have been seeing the exact same behaviour as you have outlined here.

The physical ID I see being mysteriously generated matches the same format you provided: MYSTACK-LOGICALID-B36FTLCJN97H. It took me a while to figure out where it was coming from (and I still don't really know due to the lack of documentation surrounding it.)

In my setup that reproduces this, I have several custom resources run simultaneously (they do not depend on one another). While the create handler for these custom resources are running, another resource fails to create, emitting a CREATE_FAILED. This causes the custom resources to fail with Resource creation cancelled, and this mysterious physical resource id is displayed in the physical id column. Two of the custom resources however managed to complete in time just before the CREATE_FAILED event, and did not get cancelled. These two resources correctly received the physical resource id I was expecting in the delete handler, because the create handler of the resource provider managed to finish in time. The same cannot be said about the other custom resources, the ones which had their creation cancelled received the mysterious physical id.

The custom resource delete handler I have implemented expects the physical ID passed to the delete handler to be the arn of the resource that was created (so that I can delete it). I implemented it this way because as you rightly pointed out the documentation suggests the physical ID in the delete handler will always be the value that was provided by the provider in the create event, and I made sure that this is always the case in my create handler. If a stack rollback occurs, and the custom resource creation cancels, and delete handler being called with this mysterious physical id, an error of course occurs as I implemented it to expect this value to be an ARN.

This sounds like a bug to me more than missing documentation. In some respect it doesn't make sense for a custom resource delete handler to run before a response has been put to the cloudformation response bucket by the initial create event. Otherwise, we have a situation whereby the create and delete events are being processed in parallel, and can result in very unpredictable behaviour depending on the logic the custom resource implements.

The delete handler being called before create sounds somewhat related, although probably not to these issues: https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/1730 and https://github.com/aws/aws-cdk/issues/26139

aws-cloudformation / cloudformation-coverage-roadmap