Destroying stack containing Lambda@Edge fails

[allroundexperts]

Deleting Lambda@Edge functions manually is not possible as of now in AWS. The Lambda@Edge function gets deleted automatically 1 hour after the associated Cloudfront distribution gets removed. As such, Pulumi destroy keeps on failing during that time. Ideally, Pulumi should skip the deletion of Lambda@Edge functions.

[leezen]

If I'm not mistaken, the error you're seeing above is trying to delete the actual Lambda and not the replicas. In order to delete the actual function, the replicas need to no longer exist first (which, as you state, happens after there are no associated Cloudfront distributions). So skipping probably wouldn't be the right behavior, but actually waiting for the replicas to no longer exist in order to delete the actual function would be.

In any case, it does seem to be the case that you cannot do a pulumi destroy since there's this 'invisible' replicas dependency that's not tracked, so deleting the distribution and attempting to delete the function will fail. I'll move this issue to the pulumi-aws repo to keep track.

[leezen]

As a workaround, you can pulumi state delete the function in question (which will leave it behind) if this is preventing you from destroying the rest of your stack. Then, delete the function after the replicas no longer exist.

[m4dc4p]

@leezen AWS will delete the function automatically (replicas included). So really if the SDK tracked these functions differently (and didn't attempt to delete at all) it wouldn't be a problem.

@allroundexperts did you find a workaround?

I think a simple solution Amazon could provide is to not return a 400 in this case, but just treat the call as a success (the fucntion will get deleted eventually by CloudFront anyways). If you want to show support, check out my thread on their forums https://forums.aws.amazon.com/thread.jspa?threadID=331402

[allroundexperts]

@leezen AWS will delete the function automatically (replicas included). So really if the SDK tracked these functions differently (and didn't attempt to delete at all) it wouldn't be a problem.

@allroundexperts did you find a workaround?

I think a simple solution Amazon could provide is to not return a 400 in this case, but just treat the call as a success (the fucntion will get deleted eventually by CloudFront anyways). If you want to show support, check out my thread on their forums https://forums.aws.amazon.com/thread.jspa?threadID=331402

So the only workaround that I found was using pulumi state delete the function in question. I think it would be more easier for Pulumi to treat replicas differently instead of Amazon directly.

[lukehoban]

This is being tracked upstream in https://github.com/hashicorp/terraform-provider-aws/issues/1721.

[mattiloh]

It looks like this has been fixed upstream two weeks ago with https://github.com/hashicorp/terraform-provider-aws/pull/29646. 🎉 Any ETA when this might be integrated in Pulumi? We're blocked by this issue in an e2e test-setup that creates a new stack for each build (destroying the resources manually is not a feasible option in this case).

[mattiloh]

Just tested it with the latest @pulumi/aws (v5.31.0), which seems to integrate the relevant terraform-provider-aws version (https://github.com/pulumi/pulumi-aws/releases/tag/v5.31.0). This time it tried for 10 minutes (601s) to delete two Lambda@Edge functions. Unfortunately, this might still not be enough:

aws:lambda:Function (lambda-edge-redirect-404-to-resizer):
    error: deleting urn:pulumi:test-add-e2e-tests::file-processing-service::aws:lambda/function:Function::lambda-edge-redirect-404-to-resizer: 1 error occurred:
        * deleting Lambda Function (lambda-edge-redirect-404-to-resizer-4ac83c9): operation error Lambda: DeleteFunction, https response error StatusCode: 400, RequestID: ea99fb70-6272-4b53-bad6-ba50f035cbf2, InvalidParameterValueException: Lambda was unable to delete arn:aws:lambda:us-east-1:xxx:function:lambda-edge-redirect-404-to-resizer-4ac83c9:1 because it is a replicated function. Please see our documentation for Deleting Lambda@Edge Functions and Replicas.

[H0R5E]

I'm having similar issues with a single lambda@edge function. On first try to delete it times out. If I wait a while and try again, it deletes nearly instantaneously.

A snippet of the output from the first attempt:

 -   ├─ aws:lambda:Function                 LambdaRouterFunctionHandler                                             **deleting failed**
 -   └─ aws:cloudfront:Distribution         CloudFrontDistribution                                                  deleted (235s)

Diagnostics:
  aws:lambda:Function (LambdaRouterFunctionHandler):
    error: deleting urn:pulumi:dev::sveltekit-aws-adapter-main::aws:lambda/function:Function::LambdaRouterFunctionHandler: 1 error occurred:
        * deleting Lambda Function (LambdaRouterFunctionHandler-c50e54c): operation error Lambda: DeleteFunction, https response error StatusCode: 400, RequestID: 9a414aee-c3f2-4488-83ad-de163f7204dc, InvalidParameterValueException: Lambda was unable to delete arn:aws:lambda:us-east-1:147254677601:function:LambdaRouterFunctionHandler-c50e54c:5 because it is a replicated function. Please see our documentation for Deleting Lambda@Edge Functions and Replicas.

  pulumi:pulumi:Stack (sveltekit-aws-adapter-main-dev):
    error: update failed

The second attempt wasn't instant this time as I didn't wait before calling destroy again:

 -   └─ aws:lambda:Function                 LambdaRouterFunctionHandler     deleted (148s)

[gpcherchi]

I am having the same issue and we are using @pulumi/aws (v5.40.0).

As it is fixed upstream, can we expect to get it fixed too?

[corymhall]

This is a limitation of the way Lambda@Edge works in AWS. When a Lambda@Edge function is deployed it is replicated to all of the edge locations. When it is then deleted, it has to be deleted from all of the edge locations which can take an indeterminate amount of time.

The upstream fix is as good as can be done for this issue. You can try setting a very long deletion timeout, but even then it is not guaranteed.

  new aws.lambda.Function('handler', {
    role: role.arn,
    code,
    handler: 'index.handler',
    runtime: 'nodejs20.x',
  }, {
    customTimeouts: {
      delete: '30m',
    }
  });