search

Kotaimen / awscfncli

Friendly AWS CloudFormation CLI
MIT License
59 stars 12 forks source link

Problems with cross-stack reference #69

Closed edransjsuarez closed 5 years ago

edransjsuarez commented 5 years ago

Hi,

I have the following structure into the configuration: Version: 3 Stages: Default: Vpc: StackName: AUTO-VPC Order: 1

Bastion: StackName: AUTO-Bastion Order: 2 Parameters: VPCID: ${Default.Vpc.VPCID} PublicSubnet1ID${Default.Vpc.PublicSubnet1ID}

VPCID is an output of the VPC template (the export is AUTO-VPC-VPCID). The cfn-cli is not interpolating the output. Error: 01/25/19 19:03:56 - UPDATE_ROLLBACK_IN_PROGRESS - AUTO-Bastion(AWS::CloudFormation::Stack) - Template error: Subnet ${Default.Vpc.PublicSubnet1ID} doesn't exist And: Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state Aborted!

Thanks, Jesús

Kotaimen commented 5 years ago

Hi @edransjsuarez,

Can you share the version you are using and the output with verbose option on?

cfn-cli --version
cfn-cli -v stack update

It would help us identify the issue, please remove any sensitive information form the log output (eg: account number, passwords, etc...)

GlieseRay commented 5 years ago

@edransjsuarez

I formatted your config, please also confirm that if it is right: (I noted that there is a missing comma in the last line.)

Version: 3
Stages:
  Default:
    Vpc:
      StackName: AUTO-VPC
      Order: 1

    Bastion:
      StackName: AUTO-Bastion
      Order: 2
      Parameters:
        VPCID: ${Default.Vpc.VPCID}
        PublicSubnet1ID: ${Default.Vpc.PublicSubnet1ID}
edransjsuarez commented 5 years ago

Hi @edransjsuarez,

Can you share the version you are using and the output with verbose option on?

cfn-cli --version
cfn-cli -v stack update

It would help us identify the issue, please remove any sensitive information form the log output (eg: account number, passwords, etc...)

Thanks for you response,

cfn-cli, version 2.1.4

--- end of TemplateBody ---
Capabilities: ['CAPABILITY_IAM']
Parameters: [{'ParameterKey': 'BastionAMIOS', 'ParameterValue': 'Amazon-Linux-HVM'}, {'ParameterKey': 'BastionInstanceType', 'ParameterValue': 't2.micro'}, {'ParameterKey': 'EnableBanner', 'ParameterValue': 'false'}, {'ParameterKey': 'EnableTCPForwarding', 'ParameterValue': 'true'}, {'ParameterKey': 'KeyPairName', 'ParameterValue': 'xxxkey'}, {'ParameterKey': 'NumBastionHosts', 'ParameterValue': '1'}, {'ParameterKey': 'PublicSubnet1ID', 'ParameterValue': '${Default.Vpc.PublicSubnet1ID}'}, {'ParameterKey': 'PublicSubnet2ID', 'ParameterValue': '${Default.Vpc.PublicSubnet2ID}'}, {'ParameterKey': 'RemoteAccessCIDR', 'ParameterValue': 'x.x.x.x/x'}, {'ParameterKey': 'VPCID', 'ParameterValue': '${Default.Vpc.VPCID}'}]
Tags: [{'Key': 'Stage', 'Value': 'YYY'}]
Disabling TerminationProtection
StackID: arn:aws:cloudformation:us-xxx-1:xxx:stack/YYY-Bastion/cccccc
Created: 2019-01-25 18:42:04.133000+00:00
Last Updated: 2019-01-28 19:12:35.234000+00:00
Capabilities: CAPABILITY_IAM
TerminationProtection: False
Drift Status: NOT_CHECKED
01/28/19 19:12:35 - UPDATE_IN_PROGRESS - YYY-Bastion(AWS::CloudFormation::Stack) - User Initiated
01/28/19 19:12:38 - UPDATE_ROLLBACK_IN_PROGRESS -YYY-Bastion(AWS::CloudFormation::Stack) - Template error: Subnet ${Default.Vpc.PublicSubnet1ID} doesn't exist
01/28/19 19:12:57 - UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS - YYY-Bastion(AWS::CloudFormation::Stack) - arn:aws:cloudformation:us-xxx-1:xxx:stack/YYY-Bastion/ccc
01/28/19 19:12:58 - UPDATE_ROLLBACK_COMPLETE - YYY-Bastion(AWS::CloudFormation::Stack) - arn:aws:cloudformation:us-xxx-1:XXX:stack/YYY-Bastion/ccc
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.6/site-packages/awscfncli2/cli/utils/deco.py", line 16, in wrapper
    return f(ctx, *args, **kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/awscfncli2/cli/stack/update.py", line 48, in update
    ctx.obj.runner.run(command)
  File "/home/user/.local/lib/python3.6/site-packages/awscfncli2/runner/runbook/base.py", line 48, in run
    command.run(context)
  File "/home/user/.local/lib/python3.6/site-packages/awscfncli2/runner/commands/stack_update_command.py", line 84, in run
    self.ppt.wait_until_update_complete(session, stack)
  File "/home/user/.local/lib/python3.6/site-packages/awscfncli2/cli/utils/pprint.py", line 321, in wait_until_update_complete
    waiter.wait(StackName=stack.stack_id)
  File "/home/user/.local/lib/python3.6/site-packages/botocore/waiter.py", line 53, in wait
    Waiter.wait(self, **kwargs)
  File "/home/user/.local/lib/python3.6/site-packages/botocore/waiter.py", line 323, in wait
    last_response=response,
botocore.exceptions.WaiterError: Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state

Aborted

And the complete cfn-cli.yml

Version: 3

KEY_PAIR_NAME: &KEY_PAIR_NAME
  KeyPairName: xxxkey

Stages:

  Default:

    Vpc:
      Order: 1
      StackName: YYY-VPC
      Template: templates/aws-vpc.json
      Package: True
      Region: us-xxx-1
      StackPolicy: ALLOW_ALL
      EnableTerminationProtection: false 
      Tags:
        Stage: YYY
      Parameters:
        AvailabilityZones: us-xxx-1a,us-xxx-1b,us-xxx-1c
        NumberOfAZs: 3
        InternalDomain: "example.coml"
        <<: *KEY_PAIR_NAME

    Bastion:
      Order: 2
      StackName: YYY-Bastion
      Template: templates/linux-bastion.json
      Region: us-xxx-1
      Capabilities: [CAPABILITY_IAM]
      EnableTerminationProtection: false
      Tags:
        Stage: YYY
      Parameters:
        VPCID: ${Default.Vpc.VPCID}
        PublicSubnet1ID: ${Default.Vpc.PublicSubnet1ID}
        PublicSubnet2ID: ${Default.Vpc.PublicSubnet2ID}
        RemoteAccessCIDR: x.x.x.x/x
        BastionAMIOS: Amazon-Linux-HVM
        BastionInstanceType: t2.micro
        NumBastionHosts: 1
        EnableBanner: "false"
        EnableTCPForwarding: "true"
        <<: *KEY_PAIR_NAME

Thanks

edransjsuarez commented 5 years ago

@edransjsuarez

I formatted your config, please also confirm that if it is right: (I noted that there is a missing comma in the last line.)

Version: 3
Stages:
  Default:
    Vpc:
      StackName: AUTO-VPC
      Order: 1

    Bastion:
      StackName: AUTO-Bastion
      Order: 2
      Parameters:
        VPCID: ${Default.Vpc.VPCID}
        PublicSubnet1ID: ${Default.Vpc.PublicSubnet1ID}

Hi @GlieseRay. The config I have is the following. I don't understand about where the comma is missing?

Version: 3

KEY_PAIR_NAME: &KEY_PAIR_NAME
  KeyPairName: xxxkey

Stages:

  Default:

    Vpc:
      Order: 1
      StackName: YYY-VPC
      Template: templates/aws-vpc.json
      Package: True
      Region: us-xxx-1
      StackPolicy: ALLOW_ALL
      EnableTerminationProtection: false 
      Tags:
        Stage: YYY
      Parameters:
        AvailabilityZones: us-xxx-1a,us-xxx-1b,us-xxx-1c
        NumberOfAZs: 3
        InternalDomain: "example.coml"
        <<: *KEY_PAIR_NAME

    Bastion:
      Order: 2
      StackName: YYY-Bastion
      Template: templates/linux-bastion.json
      Region: us-xxx-1
      Capabilities: [CAPABILITY_IAM]
      EnableTerminationProtection: false
      Tags:
        Stage: YYY
      Parameters:
        VPCID: ${Default.Vpc.VPCID}
        PublicSubnet1ID: ${Default.Vpc.PublicSubnet1ID}
        PublicSubnet2ID: ${Default.Vpc.PublicSubnet2ID}
        RemoteAccessCIDR: x.x.x.x/x
        BastionAMIOS: Amazon-Linux-HVM
        BastionInstanceType: t2.micro
        NumBastionHosts: 1
        EnableBanner: "false"
        EnableTCPForwarding: "true"
        <<: *KEY_PAIR_NAME
Kotaimen commented 5 years ago

Hi @edransjsuarez :

Thanks for your additional info, however I can't reproduce your issue, I uses this config (I have several production workloads running on this setup):

Version: 3

KEY_PAIR_NAME: &KEY_PAIR_NAME
  KeyPairName: <key>

TEST_ACCOUNT: &TEST_ACCOUNT
  Profile: default

Stages:

  Test:

    Order: 1

    Vpc:
      Order: 1
      StackName: AWSQuickStartVPC
      Template: https://s3.amazonaws.com/quickstart-reference/aws/vpc/latest/templates/aws-vpc.template
      Region: us-east-1
      StackPolicy: ALLOW_ALL
      EnableTerminationProtection: false
      <<: *TEST_ACCOUNT
      Tags:
        Stage: Test
      Parameters:
        AvailabilityZones: us-east-1a,us-east-1b,us-east-1c
        NumberOfAZs: 3
        NATInstanceType: t2.micro
        <<: *KEY_PAIR_NAME

    Bastion:
      Order: 2
      StackName: AWSQuickStartBastion
      Template: https://aws-quickstart.s3.amazonaws.com/quickstart-linux-bastion/templates/linux-bastion.template
      Region: us-east-1
      Capabilities: [CAPABILITY_IAM]
      EnableTerminationProtection: false
      <<: *TEST_ACCOUNT
      Tags:
        Stage: Test
      Parameters:
        VPCID: ${Test.Vpc.VPCID}
        PublicSubnet1ID: ${Test.Vpc.PublicSubnet1ID}
        PublicSubnet2ID: ${Test.Vpc.PublicSubnet2ID}
        RemoteAccessCIDR: 0.0.0.0/0
        BastionAMIOS: Amazon-Linux-HVM
        BastionInstanceType: t2.micro
        NumBastionHosts: 1
        EnableBanner: "false"
        EnableTCPForwarding: "true"
        <<: *KEY_PAIR_NAME

This is the output when I deploy Bastion stack (note the "Collect outputs"):

$cfn-cli -v -s 'Test.Bastion' stack deploy 
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
Collected outputs from Test.Vpc
Collected outputs from Test.Vpc
Collected outputs from Test.Vpc
Deploying stack Test.Bastion
StackName: AWSQuickStartBastion1
Profile: <profile>
Region: us-east-1
Account: <account>
Identity: arn:aws:sts::<account>:assumed-role/ADFS-ADMIN/bob@company.com
--- Stack Creation Parameters ---
StackName: 'AWSQuickStartBastion'
TemplateURL: 'https://aws-quickstart.s3.amazonaws.com/quickstart-linux-bastion/templates/linux-bastion.template'
Capabilities: ['CAPABILITY_IAM']
Parameters: [{'ParameterKey': 'BastionAMIOS', 'ParameterValue': 'Amazon-Linux-HVM'}, {'ParameterKey': 'BastionInstanceType', 'ParameterValue': 't2.micro'}, {'ParameterKey': 'EnableBanner', 'ParameterValue': 'false'}, {'ParameterKey': 'EnableTCPForwarding', 'ParameterValue': 'true'}, {'ParameterKey': 'KeyPairName', 'ParameterValue': '<keypairname>'}, {'ParameterKey': 'NumBastionHosts', 'ParameterValue': '1'}, {'ParameterKey': 'PublicSubnet1ID', 'ParameterValue': 'subnet-0123456781'}, {'ParameterKey': 'PublicSubnet2ID', 'ParameterValue': 'subnet-0123456782'}, {'ParameterKey': 'RemoteAccessCIDR', 'ParameterValue': '0.0.0.0/0'}, {'ParameterKey': 'VPCID', 'ParameterValue': 'vpc-0123456783'}]
Tags: [{'Key': 'Stage', 'Value': 'Test'}]
EnableTerminationProtection: False
.......

You can see in my example stack parameter get replaced during the reference step. I assume you are using AWS's Quickstart VPC with Bastion template? Please try my sample template above again and see whether your problem still persists?

edransjsuarez commented 5 years ago

Version: 3 KEY_PAIR_NAME: &KEY_PAIR_NAME KeyPairName: TEST_ACCOUNT: &TEST_ACCOUNT Profile: default Stages: Test: Order: 1 Vpc: Order: 1 StackName: AWSQuickStartVPC Template: https://s3.amazonaws.com/quickstart-reference/aws/vpc/latest/templates/aws-vpc.template Region: us-east-1 StackPolicy: ALLOW_ALL EnableTerminationProtection: false <<: TEST_ACCOUNT Tags: Stage: Test Parameters: AvailabilityZones: us-east-1a,us-east-1b,us-east-1c NumberOfAZs: 3 NATInstanceType: t2.micro <<: KEY_PAIR_NAME Bastion: Order: 2 StackName: AWSQuickStartBastion Template: https://aws-quickstart.s3.amazonaws.com/quickstart-linux-bastion/templates/linux-bastion.template Region: us-east-1 Capabilities: [CAPABILITY_IAM] EnableTerminationProtection: false <<: TEST_ACCOUNT Tags: Stage: Test Parameters: VPCID: ${Test.Vpc.VPCID} PublicSubnet1ID: ${Test.Vpc.PublicSubnet1ID} PublicSubnet2ID: ${Test.Vpc.PublicSubnet2ID} RemoteAccessCIDR: 0.0.0.0/0 BastionAMIOS: Amazon-Linux-HVM BastionInstanceType: t2.micro NumBastionHosts: 1 EnableBanner: "false" EnableTCPForwarding: "true" <<: KEY_PAIR_NAME

Hi @Kotaimen

I have the same problem running your test.

Deploying stack Test.Bastion
StackName: AWSQuickStartBastion
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
Profile: xxx
Region: us-east-1
Account: yyy
Identity: arn:aws:sts::yyy:assumed-role/xxx/xxx
--- Stack Creation Parameters ---
StackName: 'AWSQuickStartBastion'
TemplateURL: 'https://aws-quickstart.s3.amazonaws.com/quickstart-linux-bastion/templates/linux-bastion.template'
Capabilities: ['CAPABILITY_IAM']
Parameters: [{'ParameterKey': 'BastionAMIOS', 'ParameterValue': 'Amazon-Linux-HVM'}, {'ParameterKey': 'BastionInstanceType', 'ParameterValue': 't2.micro'}, {'ParameterKey': 'EnableBanner', 'ParameterValue': 'false'}, {'ParameterKey': 'EnableTCPForwarding', 'ParameterValue': 'true'}, {'ParameterKey': 'KeyPairName', 'ParameterValue': 'autokey'}, {'ParameterKey': 'NumBastionHosts','ParameterValue': '1'}, {'ParameterKey': 'PublicSubnet1ID', 'ParameterValue': '${Test.Vpc.PublicSubnet1ID}'}, {'ParameterKey': 'PublicSubnet2ID', 'ParameterValue': '${Test.Vpc.PublicSubnet2ID}'}, {'ParameterKey': 'RemoteAccessCIDR', 'ParameterValue': '0.0.0.0/0'}, {'ParameterKey': 'VPCID', 'ParameterValue': '${Test.Vpc.VPCID}'}]
Tags: [{'Key': 'Stage', 'Value': 'Test'}]
EnableTerminationProtection: False
StackID: arn:aws:cloudformation:us-east-1:xxx:stack/AWSQuickStartBastion/aaa
Created: 2019-01-31 16:38:35.131000+00:00
Capabilities: CAPABILITY_IAM
TerminationProtection: False
Drift Status: NOT_CHECKED
01/31/19 16:38:35 - CREATE_IN_PROGRESS - AWSQuickStartBastion(AWS::CloudFormation::Stack) - User Initiated
01/31/19 16:38:37 - ROLLBACK_IN_PROGRESS - AWSQuickStartBastion(AWS::CloudFormation::Stack) - Template error: Subnet ${Test.Vpc.PublicSubnet1ID} doesn't exist. Rollback requested by user.

Regards, Jesús

Kotaimen commented 5 years ago

Hi @edransjsuarez,

Hmmm.. that's strange, I can see that no parameters are extracted from the VPC stack. The parameter is parsed using Python standard librarystring.template, I'm wondering if there's difference between different Python versions... Could you also share your operation system and python version?

edransjsuarez commented 5 years ago

Hi @edransjsuarez,

Hmmm.. that's strange, I can see that no parameters are extracted from the VPC stack. The parameter is parsed using Python standard librarystring.template, I'm wondering if there's difference between different Python versions... Could you also share your operation system and python version?

Hi @Kotaimen,

Distributor ID: Ubuntu Description: Ubuntu 18.04.1 LTS Release: 18.04 Codename: bionic

Linux 4.15.0-44-generic Python 3.6.7

pip3 check awscfncli2 serverlessrepo 0.1.5 has requirement six~=1.11.0, but you have six 1.12.0. aws-sam-cli 0.10.0 has requirement six~=1.11.0, but you have six 1.12.0. aws-sam-cli 0.10.0 has requirement requests==2.20.1, but you have requests 2.21.0. launchpadlib 1.10.6 requires testresources, which is not installed.

Regards

Kotaimen commented 5 years ago

Hi @edransjsuarez,

I have identified the root cause: In formats.py:

class ParamReferenceTemplate(string.Template):
    idpattern = 'a^'
    braceidpattern = r'(?a:[_a-zA-Z]+[._a-zA-Z0-9-]*)'

Here braceidpattern is a new feature introduced in python3.7, since this is a class variable, when running on 3.6, string.Template won't complain and trying to match 'a^', which couldn't find anything to match.

@GlieseRay as its your code, I'm guessing you are trying to match ${Foo.Bar.Baz} but not $Foo.Bar.Baz here? As a short term fix I'm suggesting change pattern to explicitly match that, eg:

class ParamReferenceTemplate(string.Template):
    idpattern = r'(?a:[_a-z]+\.[_a-z0-9-]\.[_a-z0-9-])'

What do you think?

GlieseRay commented 5 years ago

@Kotaimen Agree, I will make a fix and release a hotfix version. @edransjsuarez Thank you for reporting this bug. That's very helpful

GlieseRay commented 5 years ago

@edransjsuarez Fixed in 2.1.5, please check it out