Nondescript error returned from aws cloudformation deploy

[beck3905]

I am running aws cloudformation deploy and I see in the console:

Waiting for changeset to be created..

'Status'

The return code is 255. When I look in the CloudFormation console, I see that the changeset has been created successfully, but the state is REVIEW_IN_PROGRESS. I expected this command to create the changeset and then execute it, but it seems there was an error in the process.

I did some digging in the AWS CLI source code and believe I found the issue, which I would like to report as a bug:

The reason the console shows Status is due to a KeyError occuring at:

https://github.com/aws/aws-cli/blob/648637cb5a09e71a99577a58bf967038ace2943c/awscli/customizations/cloudformation/deployer.py#L168

The cloudformation client is returning an error reporting that certain IAM permissions are missing. However, the response does not contain a Status property as expected by the line above. Instead, the response contains an Error object.

[JordonPhillips]

The response contains an error, but an exception wasn't raised? Interesting. What arguments are you giving to deploy (outside of --parameter-overrides)? What does the response look like? You can run with --debug to see.

[beck3905]

@JordonPhillips The response contained an error and an exception was raised. But a KeyError occurred in the exception handler causing the original error from the response to be hidden from the user. Instead a nondescript response is sent to the console just showing the name of the Key that is missing and not even explaining that a KeyError occurred.

[JordonPhillips]

A WaiterError should have those values filled out. What was the actual contents of the error? --debug should show you what response came over the wire.

[beck3905]

This was the response body printed to the console when using --debug:

<ErrorResponse xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/">
    <Error>
        <Type>Sender</Type>
        <Code>AccessDenied</Code>
        <Message>User: {user ARN} is not authorized to perform: cloudformation:DescribeChangeSet on resource: {resource ARN}</Message>
    </Error>
    <RequestId>e3fe78bc-3fbb-11e9-a08b-f52f3fa88d98</RequestId>
</ErrorResponse>

This is the error message from the console:

botocore.exceptions.WaiterError: Waiter ChangeSetCreateComplete failed: User: {user ARN} is not authorized to perform: cloudformation:DescribeChangeSet on resource: {resource ARN}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "{path}/lib/python3.7/site-packages/awscli/clidriver.py", line 207, in main
    return command_table[parsed_args.command](remaining, parsed_args)
  File "{path}/lib/python3.7/site-packages/awscli/clidriver.py", line 348, in __call__
    return command_table[parsed_args.operation](remaining, parsed_globals)
  File "{path}/lib/python3.7/site-packages/awscli/customizations/commands.py", line 187, in __call__
    return self._run_main(parsed_args, parsed_globals)
  File "{path}/lib/python3.7/site-packages/awscli/customizations/cloudformation/deploy.py", line 296, in _run_main
    parsed_args.fail_on_empty_changeset)
  File "{path}/lib/python3.7/site-packages/awscli/customizations/cloudformation/deploy.py", line 311, in deploy
    tags=tags
  File "{path}/lib/python3.7/site-packages/awscli/customizations/cloudformation/deployer.py", line 227, in create_and_wait_for_changeset
    self.wait_for_changeset(result.changeset_id, stack_name)
  File "{path}/lib/python3.7/site-packages/awscli/customizations/cloudformation/deployer.py", line 168, in wait_for_changeset
    status = resp["Status"]
KeyError: 'Status'
2019-03-05 19:59:42,052 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

'Status'

Status is expected to be a key in the response, not in the WaiterError object itself. Could this be an inconsistency in the API?

[beck3905]

@justnance I'm not familiar with the labels you added. Do you need more information from me? What is needed to keep this issue from being closed and lost in the ether?

[justnance]

@beck3905 - Please disregard the labels as I didn't see your response. Thanks for posting the debug output. It was helpful. This error usually indicates an issue with IAM policies or an issue with setting up permissions with the IAM role. CloudFormation updates can fail like when there is insufficient permissions to view or modify a resource.

Please compare your IAM polices to the examples in our documentation.

We have to rule this out before we continue debugging this issue.

[beck3905]

@justnance I've already fixed the IAM permissions issue so I was able to get past this. The issue I am trying to draw attention to here is the KeyError and not the underlying IAM issue.

[justnance]

@beck3905 - Thank you for clarifying. The debugs were helpful and it appear the status returns instead of:

ChangeSetCreateComplete failed: User: {user ARN} is not authorized to perform: cloudformation:DescribeChangeSet on resource: {resource ARN}

I'm labeling this as a confusing error pending further review.

[rmsy]

I also encountered this issue with the same IAM exception that @beck3905 mentioned. Fortunately their comment pointed me in the right direction and I was also able to get past the problem, but it would be nice for a useful error to be outputted.