aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.49k stars 1.17k forks source link

Allow the removal of security groups in a VPC without causing UPDATE_COMPLETE_CLEANUP_IN_PROGRESS to be stuck for 40m+ due to ENI association. #5084

Open kitsunde opened 1 year ago

kitsunde commented 1 year ago

I'm sure this isn't news, but I couldn't find an issue to track. Some time ago in my template I defined a local security group like:

NoOpSecurityGroup:
  Type: AWS::EC2::SecurityGroup

Which was associated with a function to force it to have its own ENI. Now I did an update to remove the security group, deployment goes through but it gets stuck cleaning up.

Screen Shot 2023-04-27 at 10 56 38 PM

Same issue in other frameworks:

I don't know what the solution is here, but having the builds blocked for 40+ minutes on something important makes me pretty nervous.

sriram-mv commented 1 year ago

Hey @kitsunde, thanks for opening an issue! This is behavior exposed by Cloudformation. AWS SAM is a transform on top of Cloudformation and will therefore exhibit the same performance characteristics. I will however raise this internally with the cloudformation team.

kitsunde commented 1 year ago

I understand and as far as I'm aware terraform and serverless framework also ultimately execute cloudformation, though since AWS::Serverless layers on top and aws deploy executes and has a plugin architecture I was hoping it would be possible for sam to track and destroy the ENI.

It seems like the only solution to resolve it quickly is to do aws describe-network-interfaces track down the ENI which has the security groups and then call delete-network-interface on it explicitly.

I would assume the Cloudformation team is quite aware since it has been happening for a few years, so I'm not very hopeful but thanks. :)