Custom resource not updated after dependent resource is updated (ignores DependsOn and references)

[eyalroth]

Name of the resource

AWS::CloudFormation::CustomResource

Resource Name

No response

Issue Description

I've defined a custom resource that invokes a lambda with inline code:

MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    FunctionName: myfunction
    Handler: index.handler
    Runtime: nodejs14.x
    Policies:
      - AWSLambdaVPCAccessExecutionRole
    InlineCode: |
      const cfnResponse = require('cfn-response');

      exports.handler = async function(event, context) {
        cfnResponse.send(event, context, cfnResponse.SUCCESS);
      }

MyCustomResource:
  Type: Custom::MyCustomResource
  Properties:
    ServiceToken: !GetAtt MyFunction.Arn

The resource is created successfully; however, it does not get updated by CF when the function is updated.

For instance, when I update the lambda code or add/change its timeout, the lambda gets updated but the custom resource does not (no event on the CF stack and no message in the lambda's logs).

I tried to add DependsOn and other types of references, as suggested in this SO question, but it didn't affect this behavior:

MyCustomResource:
  Type: Custom::MyCustomResource
  DependsOn: MyFunction
  Properties:
    ServiceToken: !GetAtt MyFunction.Arn
    Foo: !Ref MyFunction

I'm not 100% if this is a bug or something that is just not supported.

Expected Behavior

The custom resource should be updated and triggered with an Update event if another resource that it (directly) DependsOn is updated.

Observed Behavior

The custom resource is not updated/triggerd.

Test Cases

No response

Other Details

The larger context here is RDS schema migration/changes; every time a new schema change is added (with a code snipped), we want this change to be executed on the RDS from an AWS-based executor and not from our CI.

At first we were using a lambda to execute the migrations, but over time this proved problematic due to timeout limitations.

Therefore, we changed the DB migrations process to run from an ECS task (Fargate). For every new change we build a new image with all of the changes, push it to ECR, and update the task definition to use the new image.

What I'm trying to achieve here is to run the ECS task after it's updated as part of the CF deployment process. I wrote a lambda that will trigger the ECS task (with all the right parameters), and I want it to be triggered for every update to the ECS task definition.

Hence, I'm trying to create a custom resource that will trigger the lambda that invokes the ECS task whenever the ECS task is updated (all in the same stack/template).

[benkehoe]

This is expected behavior. CloudFormation only performs resource updates when the resource properties have changed. When your AWS::Serverless::Function resource gets updated, the ARN is not changing (nor the name, which is what !Ref MyFunction returns). So the properties on MyCustomResource don't change, so CloudFormation does not perform an update operation on that resource.

What I'm trying to achieve here is to run the ECS task after it's updated as part of the CF deployment process. I wrote a lambda that will trigger the ECS task (with all the right parameters), and I want it to be triggered for every update to the ECS task definition.

Hence, I'm trying to create a custom resource that will trigger the lambda that invokes the ECS task whenever the ECS task is updated (all in the same stack/template).

CloudFormation is generally designed to represent the graph of resources in your application, not to invoke application behavior as part of a deployment operation (like triggering a task that's defined in a stack). For that, I'd recommend placing the task ARN as an output of this stack, setting up a separate stack containing the Lambda function that triggers the task along with an SNS topic that feeds the Lambda, and then using the SNS topic ARN with the NotificationARNs property in the CreateStack API call for the application stack to send events about that stack into the SNS->Lambda you've created. It can then invoke the task when it receives the appropriate update event. https://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_CreateStack.html

(or, if the stack update is being performed as part of a CI/CD pipeline, triggering the task can be a subsequent step in the pipeline)

[eyalroth]

@benkehoe Thank you for the informative response.

CloudFormation is generally designed to represent the graph of resources in your application, not to invoke application behavior as part of a deployment operation

In our view, RDS schema migrations are not application behavior code, but rather a part of the deployment/provisioning process. They are similar to changes in the schema of a dynamo table (AWS::DynamoDB::Table).

Yes, we can invoke the migrations code via our CI/CD pipeline, but what happens if that fails? We cannot easily tell CloudFormation to rollback the stack to a state before the migrations' code was updated; this leaves the stack -- which represents a database and its schema -- in an inconsistent state.

[benkehoe]

Schema migrations are not application code per se, but what I mean is, imperative operations on the resources are different from a change to the resource graph itself.

We cannot easily tell CloudFormation to rollback the stack

I don't have a good solution here; you could wrap the CloudFormation update in a Step Function that first stashes the current template from the stack, then updates, then runs the schema migration, and on failure rolls back the stack, then ends in failure to cause the pipeline to fail.

[eyalroth]

I don't have a good solution here; you could wrap the CloudFormation update in a Step Function that first stashes the current template from the stack, then updates, then runs the schema migration, and on failure rolls back the stack, then ends in failure to cause the pipeline to fail.

@benkehoe That's a good idea, thanks!

Do you think I should close this ticket or keep it open as a feature request?

[benkehoe]

I would close it, as the specific thing in this issue is working as designed. It might be interesting to open a request for something like "synchronous post-stack-operation actions" that could cause rollback. There's a lot of ways that could be satisfied, including potentially something that would go inside the template.