RDS DatabaseInstance construct: dependency violation

[abilun]

:bug: Bug Report

What is the problem?

aws_rds.DatabaseInstance construct doesn't set dependency between RDS instance and its Security Group which causes dependency violation error while deleting the stack.

Reproduction Steps

from aws_cdk import (
    aws_ec2 as ec2,
    aws_rds as rds,
    core
)

class StorageStack(core.Stack):

    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        vpc = ec2.Vpc(
            self, "VPC",
            max_azs=2,
            cidr="172.18.0.0/16",
        )

        rds.DatabaseInstance(
            self, "RDS",
            deletion_protection=False,
            engine=rds.DatabaseInstanceEngine.POSTGRES,
            instance_identifier="myrdsinstance",
            engine_version="9.4.7",
            master_username="myuser",
            master_user_password=core.SecretValue("mypassword"),
            vpc=vpc,
            vpc_placement=vpc.select_subnets(
                subnet_type=ec2.SubnetType.PRIVATE
            ),
            instance_class=ec2.InstanceType.of(
                ec2.InstanceClass.MEMORY4, ec2.InstanceSize.LARGE),
        )

Verbose Log

Environment

• CDK CLI Version: 1.8.0 (build 5244f97)
• OS: OSX Mojave
• Language: Python 3.7.0

Other information

AWS CloudFormation StackTrace:

11:37:46 AM | DELETE_FAILED        | AWS::EC2::SecurityGroup               | RDS/SecurityGroup (RDSSecurityGroupF0B4328E) resource sg-035143f1f70d84c2a has a dependent object (Service: AmazonEC2; Status Code: 400; Error Code: DependencyViolation; Request ID: 28d7a09b-fc62-4923-8c2c-879193643fc0)

[skinny85]

Thanks for reporting @abilun . I wonder whether the error is because DatabaseInstance by default has a deletion policy of 'RETAIN', meaning it won't be removed when destroying the stack.

Can you try something like this:

rds.DatabaseInstance(
  self, 'RDS',
  # props like above...
  removal_policy=core.RemovalPolicy.DESTROY,
)

and see if that helps?

Thanks, Adam

[vgribok]

I ran into this issue using .NET CDK 1.8. I just updated RemovalPolicy and it made the failure go away. That said, this is an issue that should be detected at the synth step and not manifest itself at run time as a bad state.

[angusfz]

Ran into the same issue(CDK 1.23.0 typescript). Problem resolved by addingremovalPolicy: RemovalPolicy.DESTROY

[robertd]

Experiencing same issue here. I even extracted SG into its own stack, and declared it as a dependency of RDS stack - no luck. I'll try the workaround, but I'd like what @vgribok suggested implemented at a synth level. That would help a lot.

edit: I'm experiencing this issue using typescript.

Are you sure you want to delete: DMVectorRdsStack, DMVectorRdsSgStack (y/n)? y
DMVectorRdsStack: destroying...
   0 | 9:49:22 PM | DELETE_IN_PROGRESS   | AWS::CloudFormation::Stack                  | DMVectorRdsStack User Initiated
   0 | 9:49:24 PM | DELETE_IN_PROGRESS   | AWS::SecretsManager::SecretTargetAttachment | VectorRds/Secret/Attachment (VectorRdsSecretAttachment2E39F871)
   0 | 9:49:24 PM | DELETE_IN_PROGRESS   | AWS::CDK::Metadata                          | CDKMetadata
   1 | 9:49:25 PM | DELETE_COMPLETE      | AWS::CDK::Metadata                          | CDKMetadata
   2 | 9:49:25 PM | DELETE_COMPLETE      | AWS::SecretsManager::SecretTargetAttachment | VectorRds/Secret/Attachment (VectorRdsSecretAttachment2E39F871)
   2 | 9:49:26 PM | DELETE_SKIPPED       | AWS::RDS::DBInstance                        | VectorRds (VectorRds92A77672)
   2 | 9:49:27 PM | DELETE_IN_PROGRESS   | AWS::RDS::DBSubnetGroup                     | VectorRds/SubnetGroup (VectorRdsSubnetGroup620EE5FB)
   2 | 9:49:27 PM | DELETE_IN_PROGRESS   | AWS::SecretsManager::Secret                 | VectorRds/Secret (VectorRdsSecret66CB2648)
   3 | 9:49:28 PM | DELETE_COMPLETE      | AWS::SecretsManager::Secret                 | VectorRds/Secret (VectorRdsSecret66CB2648)
  3 Currently in progress: DMVectorRdsStack, VectorRdsSubnetGroup620EE5FB

 ❌  DMVectorRdsStack: destroy failed Error: The stack named DMVectorRdsStack is in a failed state: DELETE_FAILED (The following resource(s) failed to delete: [VectorRdsSubnetGroup620EE5FB]. )
    at /Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:165:13
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at waitFor (/Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:76:20)
    at Object.destroyStack (/Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:233:26)
    at CdkToolkit.destroy (/Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:195:9)
    at main (/Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/bin/cdk.ts:233:16)
    at initCommandLine (/Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/aws-cdk/bin/cdk.ts:168:9)
The stack named DMVectorRdsStack is in a failed state: DELETE_FAILED (The following resource(s) failed to delete: [VectorRdsSubnetGroup620EE5FB]. )
   4 | 10:33:13 PM | DELETE_FAILED        | AWS::RDS::DBSubnetGroup                     | VectorRds/SubnetGroup (VectorRdsSubnetGroup620EE5FB) Cannot delete the subnet group 'dmvectorrdsstack-vectorrdssubnetgroup620ee5fb-xr3rdziuvrrr' because at least one database instance: dv75vn8nrn60qg is still using it. (Service: AmazonRDS; Status Code: 400; Error Code: InvalidDBSubnetGroupStateFault; Request ID: 039e4c9e-f6ba-46ba-86be-77993f7a6745)
    new DatabaseInstanceNew (/Users/rdjurasaj/code/ngtoc-devops/rds/vector-rds/node_modules/@aws-cdk/aws-rds/lib/instance.ts:509:25)
    \_ new DatabaseInstanceSource (/Users/rdjurasaj/code/ngtoc-devops/rds/vector-rds/node_modules/@aws-cdk/aws-rds/lib/instance.ts:676:5)
    \_ new DatabaseInstance (/Users/rdjurasaj/code/ngtoc-devops/rds/vector-rds/node_modules/@aws-cdk/aws-rds/lib/instance.ts:785:5)
    \_ new VectorRdsStack (/Users/rdjurasaj/code/ngtoc-devops/rds/vector-rds/lib/vector-rds-stack.ts:25:24)
    \_ Object.<anonymous> (/Users/rdjurasaj/code/ngtoc-devops/rds/vector-rds/bin/vector-rds.ts:13:24)
    \_ Module._compile (internal/modules/cjs/loader.js:1151:30)
    \_ Module.m._compile (/Users/rdjurasaj/code/ngtoc-devops/rds/vector-rds/node_modules/ts-node/src/index.ts:814:23)
    \_ Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
    \_ Object.require.extensions.<computed> [as .ts] (/Users/rdjurasaj/code/ngtoc-devops/rds/vector-rds/node_modules/ts-node/src/index.ts:817:12)
    \_ Module.load (internal/modules/cjs/loader.js:1000:32)
    \_ Function.Module._load (internal/modules/cjs/loader.js:899:14)
    \_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    \_ main (/Users/rdjurasaj/code/ngtoc-devops/rds/vector-rds/node_modules/ts-node/src/bin.ts:226:14)
    \_ Object.<anonymous> (/Users/rdjurasaj/code/ngtoc-devops/rds/vector-rds/node_modules/ts-node/src/bin.ts:485:3)
    \_ Module._compile (internal/modules/cjs/loader.js:1151:30)
    \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:1171:10)
    \_ Module.load (internal/modules/cjs/loader.js:1000:32)
    \_ Function.Module._load (internal/modules/cjs/loader.js:899:14)
    \_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    \_ /Users/rdjurasaj/.nvm/versions/node/v13.9.0/lib/node_modules/npm/node_modules/libnpx/index.js:268:14
   5 | 10:33:13 PM | DELETE_FAILED        | AWS::CloudFormation::Stack                  | DMVectorRdsStack The following resource(s) failed to delete: [VectorRdsSubnetGroup620EE5FB].

[skinny85]

I believe this is a duplicate of #5282

[nija-at]

Resolving in favour of the duplicate.