Reconciliation is failing for rds aurora-postgresql database cluster and database instances

[Teja446]

What happened?

I have created a rds cluster with one reader cluster instance and one writer cluster instance using rds.aws.upbound.io/v1beta1

Note :-yaml code indentation not proper in comments , so attaching the code files as well. Please attached code aurora_xrd, aurora_comp, aurora_claim aurora_claim.txt aurora_comp.txt aurora_xrd.txt

cluster and cluster instances composition:-

apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: aurorapostgresqlcomp spec: writeConnectionSecretsToNamespace: upbound-system compositeTypeRef: apiVersion: auroradb.ccp.org/v1alpha1 kind: auroradbcluster resources:

• name: auroradbforaws base: apiVersion: rds.aws.upbound.io/v1beta1 kind: Cluster metadata: annotations: upjet.upbound.io/manual-intervention: This resource has a password secret reference. labels: env: test name: auroradbcluster spec: forProvider: engine: aurora-postgresql #patched engineVersion: "14.6" #patched

  dbClusterInstanceClass: "db.r5.2xlarge"

        storageEncrypted: true
        deletionProtection: false
        backupRetentionPeriod: 2
        preferredBackupWindow: "07:00-07:30"
        preferredMaintenanceWindow: wed:04:00-wed:04:30
        databaseName: auroradbdummy
        masterPasswordSecretRef:
          key: password
          name: sample-cluster-password
          namespace: upbound-system
        masterUsername: ccpadmin
        region: us-east-1
        #skipFinalSnapshot: true
        vpcSecurityGroupIds: 
          - sg-01b3845f9e69b5de5
          - sg-038a43b8f3765bec0
        tags:
          env: test
          db: postgresql
      writeConnectionSecretToRef:
        name: aurora-rds-cluster-secret
        namespace: upbound-system
    providerConfigRef:
      name: awsconfig

  patches:

  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.engine toFieldPath: spec.forProvider.engine
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.engineVersion toFieldPath: spec.forProvider.engineVersion
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.region toFieldPath: spec.forProvider.region
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.vpcSecurityGroupIds toFieldPath: spec.forProvider.vpcSecurityGroupIds
• name: testaurorainstance1 base: apiVersion: rds.aws.upbound.io/v1beta1 kind: ClusterInstance metadata: annotations: upjet.upbound.io/manual-intervention: This resource has a reference to Cluster, which needs manual intervention. name: auroratestinstance1 spec: forProvider: storageEncrypted: true

  preferredBackupWindow: "11:20-13:20"

      clusterIdentifierSelector:
        matchControllerRef: true
        matchLabels:
          env: test
      engine: aurora-postgresql  #patched
      instanceClass: db.r5.large #patched
      region: us-east-1 #patched
    providerConfigRef:
      name: awsconfig

  patches:

  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.dbinstanceclass toFieldPath: spec.forProvider.instanceClass
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.engine toFieldPath: spec.forProvider.engine
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.region toFieldPath: spec.forProvider.region
• name: testaurorainstance2 base: apiVersion: rds.aws.upbound.io/v1beta1 kind: ClusterInstance metadata: annotations: upjet.upbound.io/manual-intervention: This resource has a reference to Cluster, which needs manual intervention. name: auroratestinstance2 spec: forProvider: storageEncrypted: true

  preferredBackupWindow: "11:20-13:20"

      clusterIdentifierSelector:
        matchControllerRef: true
        matchLabels:
          env: test
      engine: aurora-postgresql  #patched
      instanceClass: db.r5.large #patched
      region: us-east-1 #patched
    providerConfigRef:
      name: awsconfig

  patches:

  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.dbinstanceclass toFieldPath: spec.forProvider.instanceClass
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.engine toFieldPath: spec.forProvider.engine
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.region toFieldPath: spec.forProvider.region

Error scenario 1 :-

I have created a cluster with one reader instance and one writer instance , first deleted one reader instance. Cluster instances are not getting created after deletion from console , reconciliation is failing and throwing an error “ Message: apply failed: error creating RDS Cluster (auroradbcluster-vk-lt2rm) Instance: InvalidParameterCombination: The requested DB Instance will be a member of a DB Cluster. Set backup window for the DB Cluster” . I have setup backup window for cluster , even though observing same issue

Error scenario 2 :-

After deleting one reader instance(As explained scenario 1 ) , Now remaining resources are cluster and one writer instance . Now I deleted writer instance , cluster is also getting deleted with that writer instance as there is no instance remaining . In this case also , reconciliation is not happening . Cluster and cluster instances are not getting created back . I have described the cluster claim resource to check the error , got error message as “Message: apply failed: error creating RDS cluster: InvalidParameterCombination: AllocatedStorage isn't supported for DB engine aurora-postgresql” . Allocated storage is not defined in composition itself , even though this error is observing.

How can we reproduce it? Create the cluster and cluster instance resource, for example by using the below Composition. Check the created resource READY: True and SYNCED: True Delete cluster instance and verify reconciliation (cluster instance is not creating back) Delete another instance also now cluster will also get deleted as there are no instances remaining, verify reconciliation after deleting cluster (Cluster and cluster instances are not creating back) Check the resources in rds section in aws console --resources are not getting created back , this should not happen

The XRD that we are using:

apiVersion: apiextensions.crossplane.io/v1 kind: CompositeResourceDefinition metadata: name: auroradbclusters.auroradb.ccp.org spec: group: auroradb.ccp.org names: kind: auroradbcluster plural: auroradbclusters versions:

• name: v1alpha1 served: true referenceable: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: parameters: type: object properties: engine: type: string region: type: string dbinstanceclass: type: string engineVersion: type: string required:

  • engine
  • region
  • dbinstanceclass
  • engineVersion required:
    • parameters The Composition:

  **Creates these resources:

  • DB Cluster
  • DB Instances will create and attach to DB cluster
  • VPC security groups will be attached to DB instances**

apiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: aurorapostgresqlcomp spec: writeConnectionSecretsToNamespace: upbound-system compositeTypeRef: apiVersion: auroradb.ccp.org/v1alpha1 kind: auroradbcluster resources:

• name: auroradbforaws base: apiVersion: rds.aws.upbound.io/v1beta1 kind: Cluster metadata: annotations: upjet.upbound.io/manual-intervention: This resource has a password secret reference. labels: env: test name: auroradbcluster spec: forProvider: engine: aurora-postgresql #patched engineVersion: "14.6" #patched storageEncrypted: true deletionProtection: false backupRetentionPeriod: 2 preferredBackupWindow: "07:00-07:30" preferredMaintenanceWindow: wed:04:00-wed:04:30 databaseName: auroradbdummy masterPasswordSecretRef: key: password name: sample-cluster-password namespace: upbound-system masterUsername: ccpadmin region: us-east-1 skipFinalSnapshot: true vpcSecurityGroupIds:
  • sg-01b3845f9e69b5de5
  • sg-038a43b8f3765bec0 tags: env: test db: postgresql writeConnectionSecretToRef: name: aurora-rds-cluster-secret namespace: upbound-system providerConfigRef: name: awsconfig patches:
    • type: FromCompositeFieldPath fromFieldPath: spec.parameters.engine toFieldPath: spec.forProvider.engine
    • type: FromCompositeFieldPath fromFieldPath: spec.parameters.engineVersion toFieldPath: spec.forProvider.engineVersion
    • type: FromCompositeFieldPath fromFieldPath: spec.parameters.region toFieldPath: spec.forProvider.region
    • type: FromCompositeFieldPath fromFieldPath: spec.parameters.vpcSecurityGroupIds toFieldPath: spec.forProvider.vpcSecurityGroupIds
• name: testaurorainstance1 base: apiVersion: rds.aws.upbound.io/v1beta1 kind: ClusterInstance metadata: annotations: upjet.upbound.io/manual-intervention: This resource has a reference to Cluster, which needs manual intervention. name: auroratestinstance1 spec: forProvider: storageEncrypted: true clusterIdentifierSelector: matchControllerRef: true matchLabels: env: test engine: aurora-postgresql #patched instanceClass: db.r5.large #patched region: us-east-1 #patched providerConfigRef: name: awsconfig patches:
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.dbinstanceclass toFieldPath: spec.forProvider.instanceClass
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.engine toFieldPath: spec.forProvider.engine
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.region toFieldPath: spec.forProvider.region
• name: testaurorainstance2 base: apiVersion: rds.aws.upbound.io/v1beta1 kind: ClusterInstance metadata: annotations: upjet.upbound.io/manual-intervention: This resource has a reference to Cluster, which needs manual intervention. name: auroratestinstance2 spec: forProvider: storageEncrypted: true clusterIdentifierSelector: matchControllerRef: true matchLabels: env: test engine: aurora-postgresql #patched instanceClass: db.r5.large #patched region: us-east-1 #patched providerConfigRef: name: awsconfig patches:
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.dbinstanceclass toFieldPath: spec.forProvider.instanceClass
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.engine toFieldPath: spec.forProvider.engine
  • type: FromCompositeFieldPath fromFieldPath: spec.parameters.region toFieldPath: spec.forProvider.region

The claim for this composition:

apiVersion: auroradb.ccp.org/v1alpha1 kind: auroradbcluster metadata: name: test-auroradbcluster spec: parameters: engine: aurora-postgresql engineVersion: "14.6" dbinstanceclass: db.r5.large region: us-east-1

What environment did it happen in? Crossplane Version: v1.12.1 Provider Version: v0.35.0 Kubernetes Version: v1.26.4 Kubernetes Distribution: EKS

[svscheg]

The issue reproduced on the latest master. Provider Version: v0.35.0

[Teja446]

When can we expect the fix? Any timeline decided.

[Teja446]

Reconciliation is still failing with latest version of AWS release: v0.3.6 Step1. Created Auroradb cluster with one reader and one writer instance Step2. Reconciliation verification with latest aws version : v0.36.0 :-Deleted the one reader Cluster instance((“auroradbcluster-verify-q4wfs” ), but it is not creating after deletion. Reconciliation is not working , resource is not getting created automatically.

Note :- Tried above scenarios with latest version of upbound aws provider : v0.36.0

[Teja446]

Any update on this ?

[Teja446]

Any update on fix?

[github-actions[bot]]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions[bot]]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[github-actions[bot]]

This issue is being closed since there has been no activity for 14 days since marking it as stale. If you still need help, feel free to comment or reopen the issue!