search

harrystech / arthur-redshift-etl

ELT Code for your Data Warehouse
MIT License
25 stars 11 forks source link

add deletion policy parameter #850

Closed rishi-gajula-harrys closed 11 months ago

rishi-gajula-harrys commented 11 months ago

Description

add deletion policy defaulted to retain for every resource

create change set

cloudformation/create-change-set_dw_cluster.sh dev VpcStackName=UsePreviousValue \
    MasterUsername=UsePreviousValue MasterUserPassword=UsePreviousValue \
    NodeType=UsePreviousValue NumberOfNodes=UsePreviousValue QueryConcurrency=UsePreviousValue SnapshotIdentifier=UsePreviousValue AdditionalClusterIAMRole1=UsePreviousValue AdditionalClusterIAMRole2=UsePreviousValue AdditionalClusterIAMRole3=UsePreviousValue AdditionalClusterIAMRole4=UsePreviousValue AdditionalClusterIAMRole5=UsePreviousValue AdditionalClusterIAMRole6=UsePreviousValue PreferredMaintenanceWindow=UsePreviousValue DeletionPolicyParam=Retain AdditionalSecurityGroupParam1=sg-880351fb AdditionalSecurityGroupParam2=sg-0f6ff116bbe510df9 Classic=true
cloudformation/create-change-set_dw_cluster.sh dev VpcStackName=UsePreviousValue \
    MasterUsername=UsePreviousValue MasterUserPassword=UsePreviousValue \
    NodeType=UsePreviousValue NumberOfNodes=UsePreviousValue QueryConcurrency=UsePreviousValue SnapshotIdentifier=UsePreviousValue AdditionalClusterIAMRole1=UsePreviousValue AdditionalClusterIAMRole2=UsePreviousValue AdditionalClusterIAMRole3=UsePreviousValue AdditionalClusterIAMRole4=UsePreviousValue AdditionalClusterIAMRole5=UsePreviousValue AdditionalClusterIAMRole6=UsePreviousValue PreferredMaintenanceWindow=UsePreviousValue DeletionPolicyParam=Retain AdditionalSecurityGroupParam1=sg-880351fb AdditionalSecurityGroupParam2=sg-0f6ff116bbe510df9 Classic=true
Using CloudFormation file file://./cloudformation/dw_cluster.yaml
+ STACK_PARAMETERS='ParameterKey=VpcStackName,UsePreviousValue=true ParameterKey=MasterUsername,UsePreviousValue=true ParameterKey=MasterUserPassword,UsePreviousValue=true ParameterKey=NodeType,UsePreviousValue=true ParameterKey=NumberOfNodes,UsePreviousValue=true ParameterKey=QueryConcurrency,UsePreviousValue=true ParameterKey=SnapshotIdentifier,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole1,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole2,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole3,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole4,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole5,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole6,UsePreviousValue=true ParameterKey=PreferredMaintenanceWindow,UsePreviousValue=true ParameterKey=DeletionPolicyParam,ParameterValue=Retain ParameterKey=AdditionalSecurityGroupParam1,ParameterValue=sg-880351fb ParameterKey=AdditionalSecurityGroupParam2,ParameterValue=sg-0f6ff116bbe510df9 ParameterKey=Classic,ParameterValue=true'
+ aws cloudformation validate-template --template-body file://./cloudformation/dw_cluster.yaml
+ case "$DW_VERB" in
++ date -u +%Y-%m-%d-%H-%M-%S
+ aws cloudformation create-change-set --stack-name dw-cluster-dev --change-set-name DW-2023-07-13-18-05-50 --template-body file://./cloudformation/dw_cluster.yaml --capabilities CAPABILITY_NAMED_IAM --parameters ParameterKey=VpcStackName,UsePreviousValue=true ParameterKey=MasterUsername,UsePreviousValue=true ParameterKey=MasterUserPassword,UsePreviousValue=true ParameterKey=NodeType,UsePreviousValue=true ParameterKey=NumberOfNodes,UsePreviousValue=true ParameterKey=QueryConcurrency,UsePreviousValue=true ParameterKey=SnapshotIdentifier,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole1,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole2,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole3,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole4,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole5,UsePreviousValue=true ParameterKey=AdditionalClusterIAMRole6,UsePreviousValue=true ParameterKey=PreferredMaintenanceWindow,UsePreviousValue=true ParameterKey=DeletionPolicyParam,ParameterValue=Retain ParameterKey=AdditionalSecurityGroupParam1,ParameterValue=sg-880351fb ParameterKey=AdditionalSecurityGroupParam2,ParameterValue=sg-0f6ff116bbe510df9 ParameterKey=Classic,ParameterValue=true --tags Key=user:project,Value=data-warehouse Key=user:stack-env-name,Value=dev
{
    "Id": "arn:aws:cloudformation:us-east-1:367733367673:changeSet/DW-2023-07-13-18-05-50/0586bbc6-8440-4332-8e34-dab1c46f65b2",
    "StackId": "arn:aws:cloudformation:us-east-1:367733367673:stack/dw-cluster-dev/5b028cc0-a52b-11e7-87d0-50d5cad95262"
}
+ set +x
To see resources for this stack, run:

aws cloudformation list-stack-resources --stack-name "dw-cluster-dev"

create test cluster to test if delete policy works


cloudformation/create_dw_cluster.sh delete-me VpcStackName=dw-vpc-dev \
    MasterUsername=harrys MasterUserPassword=foobarfoobar \
    NodeType=ds2.xlarge NumberOfNodes=10 QueryConcurrency=12 AdditionalClusterIAMRole1=arn:aws:iam::367733367673:role/SpectrumRawData-RedshiftGlueS3Role-dev AdditionalClusterIAMRole2=arn:aws:iam::367733367673:role/SpectrumBrazeCurrents-dev-Role AdditionalClusterIAMRole3=arn:aws:iam::367733367673:role/RedshiftSpectrumSharedGlueRole-dev AdditionalClusterIAMRole4=arn:aws:iam::367733367673:role/fivetran-reshift AdditionalClusterIAMRole5=arn:aws:iam::367733367673:role/HarrysLakeEtlTasks AdditionalClusterIAMRole6=arn:aws:iam::367733367673:role/RedshiftSpectrumCrossAccount-dev PreferredMaintenanceWindow=sat:18:30-sat:19:00 DeletionPolicyParam=Retain AdditionalSecurityGroupParam=sg-0f6ff116bbe510df9 Classic=false Environment=dev

verified that delete skip is shown in cloudformation events UI and manually deleted resource of test cluster

deleted stack

cloudformation/delete_dw_cluster.sh dev

image