aws cloudformation deploy does not clean up failed change set

[andreaswittig]

I'm using aws cloudformation deploy in a CI/CD pipeline with the option --no-fail-on-empty-changeset. This allows the pipeline to deploy all stacks, no matter if there was a change to the template or parameters.

However, aws cloudformation deploy with the option --no-fail-on-empty-changeset does create a change set which goes to state FAILED with reason The submitted information didn't contain changes. Submit different information to create a change set.. The problem is, that the failed change set does not get deleted. Therefore, hundreds of change sets pile up. Resulting in an LimitExceededException (ChangeSet limit exceeded for stack ...) after a while.

In my opinion, aws cloudformation deploy with option --no-fail-on-empty-changeset should delete change sets, that failed because of an empty change set. Or there should at least be an option to do so.

[joguSD]

I've reproduced this issue and can see how this would be less than ideal. My only concern with automatically deleting the failed change set is that it's technically a breaking change and I don't know if there are potentially customers relying on the failed change set being present.

@sanathkr Any thoughts on how to handle this?

[andreaswittig]

@joguSD Thanks for looking into this. I agree, that others might rely on the current behaviour. Although, that is not very likely. The change set does not contain any valuable information for debugging, in this scenario.

[atkinsonm]

To support users who rely on the current behavior while at the same time solving the problem for others, a new command line argument could be added (say, --cleanup-on-failure). The default behavior could remain the same as it does now. The new argument would only be relevant in cases where --no-fail-on-empty-changeset is also specified.

[mcallaghan-bsm]

There is a post explaining this in more detail fwiw: https://cloudonaut.io/aws-cli-cloudformation-deploy-limit-exceeded/

[mgorski-mg]

We had a problem with LimitExceededException so I wrote Lambda to clean it up and published it in SAR. Feel free to try! https://github.com/mgorski-mg/aws-failed-change-sets-destroyer

Of course, I'm waiting for a proper fix too! :)

[gagipro]

I encounter this issue now that we intensively use CDK with Nested Stack. everytime there's no change in a stack it creates a failed changeset in the Nested that you cannot even delete.

Only way to remove them is to create a change that updates the Nested Stack...

[Balou9]

Hi everybody.

I created a github action that might be helpful to solve this problem. https://github.com/Balou9/cfn-check-failed-status I was thinking about what you guys already mentioned: Is it wanted to have an automatic deletion of the failed change set?

For a real-world application that others rely on, maybe not. But for anything else (e.g. in the early stages of mvp programming), it might be annoying to always open the console or use the cli to delete the failed change set. At least you should have the possibility to do this automatically .

The idea is to use the action prior to the deployment step in a github actions pipeline. It checks for the stack status "FAILED" instead of failed change sets.

I would love to hear your opinion on this, especially on what needs to be improved (e.g. deleting nested stacks, check for failed change sets instead or as well as checking for failed stack status etc.)

One question that is still open for me: should this be handled by an action or an additional command line argument the aws cli?

[gagipro]

change

for the moment there is no possibility, that I'm aware of, to delete failed nested stack changeset.

so there should be this possibilty through sdk and aws cli.

[razvanphp]

This is really frustrating, please address this.

Here is a script to cleanup in bash with aws-cli: https://cloudonaut.io/aws-cli-cloudformation-deploy-limit-exceeded/

[Abhinab-AY]

Do we have any update or progress on this. This being such an important feature, it should have had much higher prio.