[Bug]: RDS instances not syncing due to wrong AZ in spec

[bobdanek]

Is there an existing issue for this?

• [X] I have searched the existing issues

Affected Resource(s)

• rds.aws.upbound.io/v1beta1 - Instance

Resource MRs required to reproduce the bug

exampledb.yml.txt mysqlinstanceandservice.yml.txt xmysqlinstance.yml.txt xmysqlinstanceandservice.yml.txt

Steps to Reproduce

I don't have a way to reproduce this outside our environment, but here's an approximation:

• Create a new RDS instance via a composite resource. Set multiAz to true. Do not specify an availability zone.
• Examine the "composed" instance (instances.rds.aws.upbound.io) to see in which AZ it landed (status.atProvider.availabilityZone)
• Examine the "composed" instance (instances.rds.aws.upbound.io) to see which AZ ends up in the spec (spec.forProvider.availabilityZone)

What happened?

Expected: RDS instance created successfully, and the instance managed resource stays synced. The AZ in which the instance was created matches the AZ that appears in the spec.

Actual behavior: RDS instance created successfully, but at some point Synced on the Instance becomes False because a different availability zone appeared in the spec, which causes replacement. Replacement is blocked (thankfully) due to "prevent_destroy":true. Any unrelated changes we want to make are blocked by this.

Relevant Error Output Snippet

conditions:
  - lastTransitionTime: "2024-06-25T16:41:47Z"
    message: 'observe failed: cannot run plan: plan failed: Instance cannot be destroyed:
      Resource aws_db_instance.example-12345-abcde has lifecycle.prevent_destroy
      set, but the plan calls for this resource to be destroyed. To avoid this error
      and continue with the plan, either disable lifecycle.prevent_destroy or reduce
      the scope of the plan using the -target flag.'
    reason: ReconcileError
    status: "False"
    type: Synced
  - lastTransitionTime: "2024-02-07T13:40:20Z"
    reason: Finished
    status: "True"
    type: AsyncOperation
  - lastTransitionTime: "2023-11-21T19:16:10Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-02-07T13:48:19Z"
    message: 'apply failed: Instance cannot be destroyed: Resource aws_db_instance.example-12345-abcde
      has lifecycle.prevent_destroy set, but the plan calls for this resource to be
      destroyed. To avoid this error and continue with the plan, either disable lifecycle.prevent_destroy
      or reduce the scope of the plan using the -target flag.'
    reason: ApplyFailure
    status: "False"
    type: LastAsyncOperation

Crossplane Version

1.14.9

Provider Version

0.40.102

Kubernetes Version

v1.28.9-eks-036c24b

Kubernetes Distribution

EKS

Additional Info

• I'm fairly new to crossplane, apologies in advance if I mix up terminology and concepts.
• All affected instances are using multi-AZ
• AZ is not specified in any of our manifests, so I'm not sure why an AZ, let alone a different AZ, is appearing in the instance managed resource
• I discovered this was occurring when trying to troubleshoot why changes to caCertIdentifier (from rds-ca-2019 to rds-ca-rsa2048-g1) were not being applied.
• We use a forked provider for feeding into our crossplane libraries: https://github.com/grafana/crossplane-provider-aws/tree/release-0.40

[bobdanek]

I found a workaround that will unblock me, though it doesn't explain why things got into this state to begin with:

If I delete availabilityZone from spec.forProvider, the resource gets updated a few seconds later, adding back availabilityZone but with the correct/expected value.

Example with kubectl:

kubectl patch instances.rds.aws.upbound.io example-db --type json -p '[{ "op": "remove", "path": "/spec/forProvider/availabilityZone" }]'