add a command to cloudformation wait any final state, and display events while waiting

[hnafar]

This is specially useful during automated deployments.

• wait until stack is in any final state:
  • CREATE_FAILED
  • DELETE_COMPLETE
  • DELETE_FAILED
  • ROLLBACK_COMPLETE
  • ROLLBACK_FAILED
  • UPDATE_COMPLETE
  • UPDATE_ROLLBACK_COMPLETE
  • UPDATE_ROLLBACK_FAILED

• log stack events emitted while waiting. Currently I achieve this using the following script, but having this as part of the cli would be much more helpful:

  
  cloudformation_tail() {
  local stack="$1"
  local region="$2"
  local lastEvent
  local lastEventId
  local stackStatus=$(aws cloudformation describe-stacks --region $region --stack-name $stack | jq -c -r .Stacks[0].StackStatus)
  
  until \
  [ "$stackStatus" = "CREATE_COMPLETE" ] \
  || [ "$stackStatus" = "CREATE_FAILED" ] \
  || [ "$stackStatus" = "DELETE_COMPLETE" ] \
  || [ "$stackStatus" = "DELETE_FAILED" ] \
  || [ "$stackStatus" = "ROLLBACK_COMPLETE" ] \
  || [ "$stackStatus" = "ROLLBACK_FAILED" ] \
  || [ "$stackStatus" = "UPDATE_COMPLETE" ] \
  || [ "$stackStatus" = "UPDATE_ROLLBACK_COMPLETE" ] \
  || [ "$stackStatus" = "UPDATE_ROLLBACK_FAILED" ]; do
  
  #[[ $stackStatus == *""* ]] || [[ $stackStatus == *"CREATE_FAILED"* ]] || [[ $stackStatus == *"COMPLETE"* ]]; do
  lastEvent=$(aws cloudformation describe-stack-events --region $region --stack $stack --query 'StackEvents[].{ EventId: EventId, LogicalResourceId:LogicalResourceId, ResourceType:ResourceType, ResourceStatus:ResourceStatus, Timestamp: Timestamp }' --max-items 1 | jq .[0])
  eventId=$(echo "$lastEvent" | jq -r .EventId)
  if [ "$eventId" != "$lastEventId" ]
  then
      lastEventId=$eventId
      echo $(echo $lastEvent | jq -r '.Timestamp + "\t-\t" + .ResourceType + "\t-\t" + .LogicalResourceId + "\t-\t" + .ResourceStatus')
  fi
  sleep 3
  stackStatus=$(aws cloudformation describe-stacks --region $region --stack-name $stack | jq -c -r .Stacks[0].StackStatus)
  done
  
  echo "Stack Status: $stackStatus"
  }

cloudformation_tail $SERVICE_STACK_NAME $REGION

which produces the following result on an update:

Awaiting completion... 2017-10-11T19:28:51.066Z - AWS::CloudFormation::Stack - my-stack-name - UPDATE_IN_PROGRESS 2017-10-11T19:28:57.591Z - AWS::ECS::TaskDefinition - TaskDefinition - UPDATE_IN_PROGRESS 2017-10-11T19:29:00.282Z - AWS::ECS::Service - Service - UPDATE_IN_PROGRESS 2017-10-11T19:36:05.919Z - AWS::CloudFormation::Stack - my-stack-name - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS Stack Status: UPDATE_COMPLETE

[jamesls]

Thanks for the suggestion, marking as a feature request.

[pcgeek86]

FYI the AWS Tools for PowerShell currently offer this feature, starting with version 3.3.119.0. The SDK includes a Wait-CFNStack command, which pauses execution until the desired CloudFormation stack state has been reached.

There's an example of its use in the blog post and documentation, linked to below.

AWS Developer Blog Post (2017-07-11) AWS Tools for PowerShell Documentation Page

Cheers, Trevor Sullivan

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[ghost]

Would be great to have this implemented. My current solution look like this

aws cloudformation deploy --stack-name "${stack_name}" \
  --template-file "${stack_template_file}" \
  --no-fail-on-empty-changeset \
  | while read -r line; do echo "$line"
      if [[ "$line" = "Waiting for stack create/update to complete" ]]; then
        "aws-cloudformation-deploy-watch.sh" --stack-name "${stack_name}"
      fi
    done

aws-cloudformation-deploy-watch.sh

#!/usr/bin/env sh -euo pipefail

############################## Usage ################################
#
# aws-cloudformation-deploy-watch.sh --stack 'STACK_NAME'
#
#####################################################################

while [ $# -gt 0 ]; do
  case $1 in
    --stack-name) stack_name_opt="$2"; shift 2;;    
    --profile)    profile_opt="$2"; shift 2 ;;
    --region)     region_opt="$2"; shift 2;;
    *) echo "$0: unexpected argument: $1" >&2; exit 1;;
  esac
done

_TAB=$'\t'
most_recent_event_id='???'
while : ; do
  # get recent stack events
  most_recent_event_id_sed_escaped="$(echo "$most_recent_event_id" | sed -e 's/[]\/$*.^[]/\\&/g')"
  recent_events="$(aws cloudformation describe-stack-events \
      $([ -n "${stack_name_opt:-}" ] && echo "--stack-name $stack_name_opt") \
      --query 'StackEvents[*].[EventId,Timestamp,LogicalResourceId,ResourceStatus,ResourceStatusReason]' \
      $([ -n "${profile_opt:-}" ] && echo "--profile $profile_opt") \
      $([ -n "${region_opt:-}" ] && echo "--region $region_opt") \
      --output text \
    | sed  "/${_TAB}User Initiated\$/q" | sed "/^${most_recent_event_id_sed_escaped}${_TAB}/,\$d" \
    | tail -r)"
  if [ -n "$recent_events" ]; then
    # save last event id
    most_recent_event_id=$(echo "$recent_events" | tail -1 | awk -F $'\t' '{printf $1}')
    while read -r event; do
      # print new event
      echo "$event" | awk -F $'\t' '{printf "%-28s %-48s %-24s %s\n",$2,$3,$4,$5}' | sed "s/None$//"
      # looking for stack resource events
      event_resouce=$(echo "$event" | awk -F $'\t' '{printf $3}')
      if [ "$event_resouce" == "$stack_name_opt" ]; then
        stack_status="$(echo "$event" | awk '{printf $4}')"
        # exit on final stack status
        if [[ "$stack_status" = *'_FAILED' ]] || [[ "$stack_status" = *'_ROLLBACK_COMPLETE' ]]; then exit 1; fi
        if [[ "$stack_status" = *'_COMPLETE' ]]; then exit 0; fi
      fi
    done <<< "$recent_events"
  fi
  sleep 1
done

[thearthur]

Can we re-open this here since the uservoice migration appears no longer to be a thing? (or at least the links here are broken)

[lantrix]

It seems this has been implemented here: https://docs.aws.amazon.com/cli/latest/reference/cloudformation/wait/index.html

However you need to call different operations with the CLI to differentiate between waiting for a Stack Create vs. a Stack Update to be complete:

aws cloudformation wait stack-create-complete --stack-name mystack
aws cloudformation wait stack-update-complete --stack-name mystack

The Powershell implementation of this (Wait-CFNStack) uses one command to hit DescribeStacks API and wait for the status of your choice:

Wait-CFNStack -StackName mystack -Timeout 300 -Status CREATE_COMPLETE,UPDATE_COMPLETE

It's a bit of a strange situation though. CLI and Powershell SDK usage are flipped in the way they work.

With the CLI I can use aws cloudformation deploy (creating and then executing a change set) which will update a stack or create a new stack with one command. Great for my CD that doesn't need to know if it is a first time deploy or an update. However I then have to determine if the stack was created OR updated to pick one of the two AWS CLI commands:

• CREATE_COMPLETE aws cloudformation wait stack-create-complete
• UPDATE_COMPLETE aws cloudformation wait stack-update-complete

However with PowerShell SDK I can also use the DescribeStacks API to determine if the stack was created OR updated, however I get a single cmdlet I can use Wait-CFNStack. Great for CD - HOWEVER I'm unable to have a single command to create or update my stack. I have to pick one or the other.

• New-CFNStack - the CreateStack API operation
  • New-CFNStack -StackName "myStack" -TemplateBody "{TemplateContent}"
• Update-CFNStack -the UpdateStack API operation
  • Update-CFNStack -StackName "myStack" -TemplateBody "{TemplateContent}"

I've written PowerShell to detect if the stack exists, and do change sets - but this is onerous.

[revmischa]

This feature would be super useful for scripting test deployments of my cloudformation stack in CI

[weickmanna]

+1

[weickmanna]

+1

[mattBrzezinski]

Would love to see, any updates on this feature request?

[mdforbes500]

Any progress on this? It is really annoying to have to specify in jenkins whether you need an update or creation.

[jmclean-cnexus]

+1

[3lricko]

+1

[covertbert]

Are there any plans to implement this? It's crazy that we all have to write scripts ourselves to do this (and they always end up being a little flakey in my experience). This command would be more useful than any of the individual "wait for a specific state" commands that already exist.

[tim-finnigan]

These are the wait commands currently available for cloudformation:

• change-set-create-complete
• stack-create-complete
• stack-delete-complete
• stack-exists
• stack-import-complete
• stack-rollback-complete
• stack-update-complete
• type-registration-complete

(https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudformation/wait/index.html)

A few different topics have been mentioned here in the discussion so I want to confirm what the feature request is. @hnafar I believe you’re asking for the ability to see every stack status logged while waiting to reach one of those states above. Is that correct?

[dfehrenbach]

I don't mean to speak for @hnafar, but it's the idea that "wait commands" should be allowed to be bundled. Either do that by saying:

1. "Any one of the final states have been reached."
2. "Any one of the final states that you have defined have been reached."

As an example, imagine a ci/cd pipeline where you want to auto-update your cloud formation stack when you've pushed new code. It's a simple idea, but to avoid errors from the AWS CLI, you must define a series of waits based on the current status of the stack.

1. Is it not created yet? Well, you wait create-stack-complete before using either create-stack or deploy.
2. Is it created but currently updating? Then, you wait update-stack-complete.
3. Was the stack actually rolling back and you're pushing a quick change to fix whatever caused the rollback? Then, you wait stack-rollback-complete. etc.

This is not to mention that it's not clear what happens mid update when your stack has to roll back, but you're in the middle of wait update-stack-complete since your code inspected the status and at the time the cf stack was mid-update.

Simply, it's cumbersome. I'd not say it's all bad, however. The wait clauses being there are far better than the alternative (but that's neither here nor there).

Here's an example implementation (still a bit limited) that collects final states: https://www.npmjs.com/package/@cumulusds/aws-cloudformation-wait-ready It uses just a simple string match on the end to inspect for final states: (i.e. https://github.com/CumulusDS/aws-cloudformation-wait-ready/blob/master/src/index.js#L22 )

Another implementation to look at would be the aforementioned one in PowerShell (as called out by @pcgeek86 & @lantrix). Doing this but in AWS CLI is what I would think the goal is. I imagine a day where we're done in 2 lines (please don't take the following as a concrete task, it's an expression of thought and is simply for ideation):

aws cloudformation wait cfn-stack --status CREATE_COMPLETE,UPDATE_COMPLETE ... (new command) aws cloudformation deploy ... (existing command)

Or, what if one line? aws cloudformation deploy --wait-for-status CREATE_COMPLETE,UPDATE_COMPLETE ...

[covertbert]

While I also don't claim to speak for @hnafar, what @dfehrenbach says describes my issue exactly.

We also use the npm package aws-cloudformation-wait-ready but it's not perfect at all and sometimes we still have issues where that script will be "done waiting" but when the next step in the CI pipeline (a cloudformation update) runs it will still error due to the stack still creating/updating/rollbacking/whatever.

The usefulness of this feature for CI pipelines cannot be overstated IMO.