CloudFormation - Create / Update stacks dose not surface stack errors in logs

[ghost]

Describe the bug When the Cloudformation create / update stack task fails the API response with the offending Logical ID and Status reason is not surfaced in the task logs. it is not possible to debug the error from the release and requires the user to retrieve the errors direct from Cloudformation manually.

To reproduce create a cfn template with an error and deploy using the cloud formation create stack task. the the task is run and fails. the actual error is not surfaced. the task logs only states that the task is in the wrong state.

Expected behavior create a cfn template with an error and deploy using the cloud formation create stack task. the the task is run and fails. the Logical ID and the Status reason will be printed to the logs.

Your Environment

• Azure Devops cloud based

[hunterwerlla]

I'm working on refactoring this module right now, so I'll see if it's an easy fix for the next release

[ghost]

Awesome cheers dude!

On Fri, 17 May 2019 at 17:08, Hunter Werlla notifications@github.com wrote:

I'm working on refactoring this module right now, so I'll see if it's an easy fix for the next release

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aws/aws-vsts-tools/issues/199?email_source=notifications&email_token=AKBPQZR6VLUUYRZ232AIXXDPV3J75A5CNFSM4HNQXY6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVVFWVY#issuecomment-493509463, or mute the thread https://github.com/notifications/unsubscribe-auth/AKBPQZWYBWWWZ4XZ7ZTQ2K3PV3J75ANCNFSM4HNQXY6A .

--

Steven Cuthill DevOps Manager

steven.cuthill@onthemarket.com www.onthemarket.com

Download the OnTheMarket.com app...

https://itunes.apple.com/gb/app/onthemarket.com-property-search/id960416200?mt=8 https://play.google.com/store/apps/details?id=com.onthemarket.mobile&hl=en_GB

Follow us on...

https://www.facebook.com/Onthemarketcom-1500133890261960/ https://twitter.com/OnTheMarketCom https://www.linkedin.com/company/onthemarket

[hunterwerlla]

I tried a minimal case, where there is an obvious dependency loop, and the error was surfaced correctly in the build log as a circular dependency error. What category was the error?

[ghost]

Hi @hunterwerlla,

sorry for the long delay at getting this, wanted to generate a real world example,

so i have made some changes to one of our CloudFormation stacks via the cfn tasks and it looks like i have made a mistake and the error is highlighted in the aws cfn console.

2019-06-24 14:36:13 UTC+0100	internal-prd-eu-west-2-ecsCluster	UPDATE_ROLLBACK_IN_PROGRESS	No export named internal-prd-eu-west-2-iam-AzurePipelineAgentTaskRole found	AWS::CloudFormation::Stack

However the error message in the logs from the cfn task is not surfaced, we just get the Error: Stack internal-prd-eu-west-2-ecsCluster failed to reach update completion status, error: 'Resource is not in the state stackUpdateComplete' and thats not very helpful, you end up having to always refer back to the aws console to find out what went wrong.

The task may also require permissions to upload your application template to the specified Amazon S3 bucket.
2019-06-24T13:35:40.0307203Z ==============================================================================
2019-06-24T13:35:40.6757713Z Configuring credentials for task
2019-06-24T13:35:40.6761615Z e3b81a72-c761-4a6a-8373-e87825107277 exists true
2019-06-24T13:35:40.6762375Z ...configuring AWS credentials from service endpoint 'e3b81a72-c761-4a6a-8373-e87825107277'
2019-06-24T13:35:40.6762529Z ...endpoint defines standard access/secret key credentials
2019-06-24T13:35:40.6807979Z Configuring region for task
2019-06-24T13:35:40.6809582Z ...configured to use region eu-west-2, defined in task.
2019-06-24T13:35:40.6902848Z Configuring credentials for task
2019-06-24T13:35:40.6906074Z e3b81a72-c761-4a6a-8373-e87825107277 exists true
2019-06-24T13:35:40.6906445Z ...configuring AWS credentials from service endpoint 'e3b81a72-c761-4a6a-8373-e87825107277'
2019-06-24T13:35:40.6907132Z ...endpoint defines standard access/secret key credentials
2019-06-24T13:35:40.6907421Z Configuring region for task
2019-06-24T13:35:40.6909244Z ...configured to use region eu-west-2, defined in task.
2019-06-24T13:35:40.7040252Z Checking existence for stack internal-prd-eu-west-2-ecsCluster
2019-06-24T13:35:40.9001881Z Stack exists, switching to update stack mode
2019-06-24T13:35:40.9008749Z Testing to see if change set GitCommit-87112d244949cd6f12618cf6d320f235e7136a35 associated with stack internal-prd-eu-west-2-ecsCluster exists
2019-06-24T13:35:41.0074603Z Test for existence of change set GitCommit-87112d244949cd6f12618cf6d320f235e7136a35 returned error: 'ChangeSet [GitCommit-87112d244949cd6f12618cf6d320f235e7136a35] does not exist'.
2019-06-24T13:35:41.0085158Z Uploading template file d:\a\r1\a\internal-infrastructure\v1.387\cloudFormation\stackTemplates\ecsClusters.cf.json to object ecsClusters.cf.json in bucket internal-prd-eu-west-2-cloudformation/v1.387/Release-167
2019-06-24T13:35:41.1337076Z Generated url to template: https://internal-prd-eu-west-2-cloudformation.s3.eu-west-2.amazonaws.com/v1.387/Release-167/ecsClusters.cf.json?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=***%2F20190624%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20190624T133541Z&X-Amz-Expires=900&X-Amz-Signature=58338e0df84d8baa9ded4995aaaa150c32a70c5ddb548d9bb0bd0327feb64687&X-Amz-SignedHeaders=host
2019-06-24T13:35:41.1339065Z Loading template parameters file 'd:\a\r1\a\internal-infrastructure\v1.387\cloudFormation\stackParameters\prd\ecsClusters.json'
2019-06-24T13:35:41.1342602Z Successfully loaded template parameters
2019-06-24T13:35:41.1344747Z Setting capability CAPABILITY_IAM for stack
2019-06-24T13:35:41.1344991Z Setting capability CAPABILITY_NAMED_IAM for stack
2019-06-24T13:35:41.1346340Z Creating UPDATE type change set GitCommit-87112d244949cd6f12618cf6d320f235e7136a35
2019-06-24T13:35:41.3269008Z Waiting for change set GitCommit-87112d244949cd6f12618cf6d320f235e7136a35 to be validated for stack internal-prd-eu-west-2-ecsCluster
2019-06-24T13:36:11.5738878Z Change set successfully validated
2019-06-24T13:36:11.5739287Z Executing change set GitCommit-87112d244949cd6f12618cf6d320f235e7136a35 internal-prd-eu-west-2-ecsCluster
2019-06-24T13:36:11.8453620Z Waiting for stack internal-prd-eu-west-2-ecsCluster to reach update complete status
2019-06-24T13:36:42.0780568Z Change set execution request failed with error: 'Stack internal-prd-eu-west-2-ecsCluster failed to reach update completion status, error: 'Resource is not in the state stackUpdateComplete'' Error: Stack internal-prd-eu-west-2-ecsCluster failed to reach update completion status, error: 'Resource is not in the state stackUpdateComplete'
2019-06-24T13:36:42.0781203Z     at Object.<anonymous> (d:\a\_tasks\CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99\1.4.0\CloudFormationCreateOrUpdateStack.js:13:1644)
2019-06-24T13:36:42.0781306Z     at throw (native)
2019-06-24T13:36:42.0781418Z     at n (d:\a\_tasks\CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99\1.4.0\CloudFormationCreateOrUpdateStack.js:13:164)
2019-06-24T13:36:42.0781506Z     at process._tickDomainCallback (internal/process/next_tick.js:135:7)
2019-06-24T13:36:42.0781737Z Change set creation request failed with error: 'Stack internal-prd-eu-west-2-ecsCluster failed to reach update completion status, error: 'Resource is not in the state stackUpdateComplete'' Error: Stack internal-prd-eu-west-2-ecsCluster failed to reach update completion status, error: 'Resource is not in the state stackUpdateComplete'
2019-06-24T13:36:42.0782016Z     at Object.<anonymous> (d:\a\_tasks\CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99\1.4.0\CloudFormationCreateOrUpdateStack.js:13:1644)
2019-06-24T13:36:42.0782235Z     at throw (native)
2019-06-24T13:36:42.0782440Z     at n (d:\a\_tasks\CloudFormationCreateOrUpdateStack_7ef7cdfa-aa45-42c5-93c8-d7603643dd99\1.4.0\CloudFormationCreateOrUpdateStack.js:13:164)
2019-06-24T13:36:42.0782607Z     at process._tickDomainCallback (internal/process/next_tick.js:135:7)

Generally i think the output needs to be improved, it should surface any errors at a stack level ( validation error ) and any resource level errors. ideally having the same level out output as the CDK would be a lot better.

[ghost]

@hunterwerlla any any updated on this ?

[ghost]

@hunterwerlla any plans to add this in to the project ?

[hunterwerlla]

CloudFormation output (and task output in general) is one of the areas I want to look at improving, but I do not currently have a timeline for when the work will be done