'aws cloudformation wait stack-update-complete' Not Working

[pit]

Hello!

I'm running aws cloudformation wait stack-update-complete command and with or without --stack-name argument it fails with following error:

peter@yoga:~$ aws cloudformation wait stack-update-complete Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state

Stack trace after --debug switch:

2016-07-10 09:53:15,847 - MainThread - botocore.hooks - DEBUG - Event needs-retry.cloudformation.DescribeStacks: calling handler <botocore.retryhandler.RetryHandler object at 0x7f8d0340dc50> 2016-07-10 09:53:15,847 - MainThread - botocore.retryhandler - DEBUG - No retry needed. 2016-07-10 09:53:15,848 - MainThread - awscli.clidriver - DEBUG - Exception caught in main() Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 186, in main return command_table[parsed_args.command](remaining, parsed_args) File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 379, in call return command_table[parsed_args.operation](remaining, parsed_globals) File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/commands.py", line 190, in call parsed_globals) File "/usr/local/lib/python2.7/dist-packages/awscli/clidriver.py", line 549, in call call_parameters, parsed_globals) File "/usr/local/lib/python2.7/dist-packages/awscli/customizations/waiters.py", line 206, in invoke waiter.wait(_parameters) File "/usr/local/lib/python2.7/dist-packages/botocore/waiter.py", line 53, in wait Waiter.wait(self, _kwargs) File "/usr/local/lib/python2.7/dist-packages/botocore/waiter.py", line 318, in wait reason='Waiter encountered a terminal failure state') WaiterError: Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state 2016-07-10 09:53:15,854 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state

AWS CLI version

peter@yoga:~$ aws --version aws-cli/1.10.46 Python/2.7.12 Linux/4.4.0-30-generic botocore/1.4.36

[kyleknap]

Could you possibly include the final response from DescribeStack from your debug log? I would need that to determine what terminal failure state was hit and explain why you are seeing this.

[JordonPhillips]

Closing due to inactivity

[pit]

Sorry for not replying here, was out of city for a long time. So, error still occurs. As for DescribeStack response - I have this error without specifying stack-name attribute, so it's reproducible without this info. It's up to botocore issue I suppose, but it's only my guess.

[pit]

I think I've got wy you need DescribeStack. Here it is: https://gist.github.com/pit/cdc298109088318dc497af7f0b28b419

[pit]

@JordonPhillips reopen please

[JordonPhillips]

@pit It looks like your latest update had to rollback, which is a failure state for that waiter.

[pit]

I see, ok. Specifying stack id helps me and waiter works ok for that case.

But why does it failure state for waiter in case no stacks are given?

[JordonPhillips]

Honestly I can't imagine why it would work at all without an id specified. In this case specifically it is failing because it sees a failure state in the first response.

[pit]

Ok. As for me I have to use --stack-name argument and this is enough for the moment. But could you please investigate and fix this issue with running without arguments?

[dennis-bell]

I am experiencing the same issue, any progress?

[stealthycoin]

Closing due to inactivity. Also, there is no use case for calling it with no arguments as it has nothing to wait on.

[andrewmyhre]

I'm finding that if my stack is in the UPDATE_ROLLBACK_FAILED state and I invoke aws cloudformation continue-update-rollback --stack-name my-stack and subsequently invoke aws cloudformation wait stack-update-complete --stack-name my-stack I get a Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state back from the cli even when the stack successfully rolls back. It seems that wait stack-update-complete isn't the correct call when rolling back an update. Is that correct? In our case we want to follow this with another aws cloudformation create-change-set call and I want to make sure the stack is in a good state before I do so, so I want to use an aws cloudformation wait command in the interim.

Many thanks

[GeniusWiki]

@andrewmyhre This is still true... I am facing the same issue. stack-update-complete doesn't work if stack status is UPDATE_ROLLBACK_COMPLETE

[raonitimo]

@andrewmyhre This is still true... I am facing the same issue. stack-update-complete doesn't work if stack status is UPDATE_ROLLBACK_COMPLETE

From the documentation, the command will wait for an UPDATE_COMPLETE, which is a success state. UPDATE_ROLLBACK_COMPLETE is an error state.

So, this is the expected behavior.

I'm finding that if my stack is in the UPDATE_ROLLBACK_FAILED state and I invoke aws cloudformation continue-update-rollback --stack-name my-stack and subsequently invoke aws cloudformation wait stack-update-complete --stack-name my-stack I get a Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state back from the cli even when the stack successfully rolls back. It seems that wait stack-update-complete isn't the correct call when rolling back an update. Is that correct? In our case we want to follow this with another aws cloudformation create-change-set call and I want to make sure the stack is in a good state before I do so, so I want to use an aws cloudformation wait command in the interim.

Many thanks

Maybe you want a wait for UPDATE_ROLLBACK_COMPLETE, like running: aws cloudformation wait stack-update-rollback-complete --stack-name my-stack ? Does it make any sense?

[MatthewEdge]

Is it fair to say that use of the aws cloudformation wait stack-update-complete command as a means of preventing parallel runs is an anti-pattern?

Using it for this purpose works for us until we hit a rollback scenario. The only recourse seems to be to go in manually and update the stack to get back to UPDATE_COMPLETE. Only then can subsequent builds using the command is this fashion can proceed.

Is there an alternative that should be used for this purpose?

[raonitimo]

Is it fair to say that use of the aws cloudformation wait stack-update-complete command as a means of preventing parallel runs is an anti-pattern?

Using it for this purpose works for us until we hit a rollback scenario. The only recourse seems to be to go in manually and update the stack to get back to UPDATE_COMPLETE. Only then can subsequent builds using the command is this fashion can proceed.

Is there an alternative that should be used for this purpose?

For preventing parallel runs, I'd suggest running a describe-stacks and checking if the StackStatus is one of these SUCCESS status [1] before performing any update: CREATE_COMPLETE, ROLLBACK_COMPLETE, UPDATE_COMPLETE, UPDATE_ROLLBACK_COMPLETE.

So, in the event of a rollback, it wouldn't be necessary to perform manual updates.

References: [1] Describing and Listing Your Stacks - - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-describing-stacks.html#w2ab1c15c15c17c11b4