search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.51k stars 1.17k forks source link

Unable to deploy nested SecretsManagerRDSOracleRotationSingleUser app #1684

Closed dinvlad closed 4 years ago

dinvlad commented 4 years ago

Description

Unable to deploy a nested application (SecretsManagerRDSOracleRotationSingleUser).

Steps to reproduce

  1. Prepare template.yaml with the following resource: 
    DBSecretRotationApp:
Type: AWS::Serverless::Application
Properties:
  Location:
    ApplicationId: arn:aws:serverlessrepo:us-east-1:297356227824:applications/SecretsManagerRDSOracleRotationSingleUser
    SemanticVersion: 1.1.7
  Parameters:
    endpoint: !Sub https://secretsmanager.${AWS::Region}.amazonaws.com
    functionName: OracleSecretRotationLambda
  2. Deploy with: 
    sam deploy \
--stack-name test-stack \
--capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \
--debug

Observed result

Stack creation fails with a message similar to the following:

Embedded stack arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:stack/yyyy-DBSecretRotationApp-zzzz/aaaa was not successfully created: The following resource(s) failed to create: [DBSecretRotationApp].

and no other details. Output of sam deploy with --debug:

Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics 

    Deploying with following values 
    =============================== 
    Stack name                 : <stack_name>
    Region                     : None 
    Confirm changeset          : False 
    Deployment s3 bucket       : None 
    Capabilities               : ["CAPABILITY_IAM", "CAPABILITY_AUTO_EXPAND"] 
    Parameter overrides        : <...>

Initiating deployment 
===================== 
Property Location of DBSecretRotationApp resource is not a URL 

Waiting for changeset to be created.. 

CloudFormation stack changeset 
------------------------------------------------------------------------------------------------ 
Operation                        LogicalResourceId                ResourceType                    
------------------------------------------------------------------------------------------------ 
+ Add                            DBSecretRotationApp              AWS::CloudFormation::Stack      
------------------------------------------------------------------------------------------------ 

Changeset created successfully. arn:aws:cloudformation:us-east-1:xxxxxxxxxxxx:changeSet/samcli-deploy1234/abcd

2019-12-18 18:05:44 - Waiting for stack create/update to complete 

CloudFormation events from changeset 
------------------------------------------------------------------------------------------------- 
ResourceStatus           ResourceType             LogicalResourceId        ResourceStatusReason    
------------------------------------------------------------------------------------------------- 
CREATE_IN_PROGRESS       AWS::CloudFormation::S   DBSecretRotationApp      -                       
                         tack                                                                      
CREATE_IN_PROGRESS       AWS::CloudFormation::S   DBSecretRotationApp      Resource creation       
                         tack                                              Initiated               
CREATE_FAILED            AWS::CloudFormation::S   DBSecretRotationApp      Embedded stack arn:aws  
                         tack                                              :cloudformation:us-eas  
                                                                           t-1:xxxxxxxxxxxx:stack  
                                                                           /<stack_name>-DBSecretR  
                                                                           otationApp-zzzz/aaaa was    
                                                                           not successfully        
                                                                           created: The following  
                                                                           resource(s) failed to   
                                                                           create: [SecretsManage  
                                                                           rRDSOracleRotationSing  
                                                                           leUser].                
UPDATE_ROLLBACK_IN_PRO   AWS::CloudFormation::S   <stack_name>             The following           
GRESS                    tack                                              resource(s) failed to   
                                                                           create:                 
                                                                           [DBSecretRotationApp].  
DELETE_IN_PROGRESS       AWS::CloudFormation::S   DBSecretRotationApp      -                       
                         tack                                                                      
UPDATE_ROLLBACK_COMPLE   AWS::CloudFormation::S   <stack_name>              -                       
TE_CLEANUP_IN_PROGRESS   tack                                                                      
UPDATE_ROLLBACK_COMPLE   AWS::CloudFormation::S   <stack_name>              -                       
TE                       tack                                                                      
DELETE_COMPLETE          AWS::CloudFormation::S   DBSecretRotationApp      -                       
                         tack                                                                      
------------------------------------------------------------------------------------------------- 
Execute changeset waiter exception 
Traceback (most recent call last): 
  File "/usr/local/lib/python3.8/site-packages/samcli/lib/deploy/deployer.py", line 395, in wait_for_execute 
    waiter.wait(StackName=stack_name, WaiterConfig=waiter_config) 
  File "/usr/local/lib/python3.8/site-packages/botocore/waiter.py", line 53, in wait 
    Waiter.wait(self, **kwargs) 
  File "/usr/local/lib/python3.8/site-packages/botocore/waiter.py", line 320, in wait 
    raise WaiterError( 
botocore.exceptions.WaiterError: Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state 
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam deploy', 'duration': 89551, 'exitReason': 'DeployFailedError', 'exitCode': 1, 'requestId': 'xyz', 'installationId': '123', 'sessionId': '456', 'executionEnvironment': 'CLI', 'pyversion': '3.8.0', 'samcliVersion': '0.38.0'}}]} 
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1) 
Error: Failed to create/update the stack: <stack_name>, Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state

Expected result

Stack should be deployed successfully. The application is an official AWS app for rotating RDS secrets. Perhaps the culprit is that it requires CAPABILITY_RESOURCE_POLICY? sam deploy --capabilities ... does not recognize this capability as a valid value.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: aws/codebuild/standard:3.0
  2. sam --version: SAM CLI, version 0.38.0
dinvlad commented 4 years ago

It was a permissions issue. I was able to troubleshoot it by looking at the nested stack, thanks to https://www.reddit.com/r/aws/comments/94lb5z/cloudformation_troubleshooting/. It would be helpful if SAM/CFN printed nested errors so we didn't have to know where exactly to look..

awood45 commented 4 years ago

Unfortunately there isn't much we can do beyond what you were able to get with the debug output - can pass this feedback on to the CloudFormation team, which provides the status messages we rely on.

dinvlad commented 4 years ago

Thanks, would much appreciate that!