ACK RDS Controller: performanceInsightsEnabled manual changes had different behavior

Describe the bug We have a global RDS cluster with primary and secondary clusters, where the writer and reader instances are managed by the ACK RDS Controller.

Recently, one of our DBAs manually enabled Performance Insights on the writer instances using the AWS Console. Shortly afterward, we noticed that the changes were reflected in the Kubernetes cluster, with the updated values being propagated in both the spec and status fields of the DBInstance object.

This behavior was unexpected, as we assumed that the source of truth would be the ACK object in the Kubernetes cluster, and that any manual changes made directly in AWS would not be automatically reflected in the Kubernetes objects.

Below is a snippet of the DBInstance object that shows the propagated values:

  dbName: xxx
  deletionProtection: false
  engine: aurora-mysql
  engineVersion: 8.0.mysql_aurora.3.02.0
  kmsKeyID: arn:aws:kms:us-west-2:xxx:key/xxx
  licenseModel: general-public-license
  masterUsername: root
  monitoringInterval: 0
  monitoringRoleARN: arn:aws:iam::xxxx:role/xxx
  multiAZ: false
  networkType: IPV4
  performanceInsightsEnabled: false
  preferredBackupWindow: 09:36-10:06
  preferredMaintenanceWindow: fri:10:50-fri:11:20
  promotionTier: 1
  publiclyAccessible: false
  storageEncrypted: true
  storageThroughput: 0
  storageType: aurora
status:
  ackResourceMetadata:
    arn: arn:aws:rds:us-west-2:xxx:db:us-west-2-xxx-db-instance-0
    ownerAccountID: "xxx"
    region: us-west-2
  certificateDetails:
    cAIdentifier: rds-ca
    validTill: "2025-07-19T06:31:47Z"
  conditions:
  - message: "AccessDenied: User: arn:aws:sts::xxx:assumed-role/xxx-ack-controller/xxxx
      is not authorized to perform: iam:PassRole on resource: arn:aws:iam::xxxx:role/rds-monitoring-role
      because no identity-based policy allows the iam:PassRole action\n\tstatus code:
      403, request id: xxx"
    status: "True"
    type: ACK.Recoverable
  - lastTransitionTime: "2024-08-01T14:47:54Z"
    message: Unable to determine if desired resource state matches latest observed
      state
    reason: "AccessDenied: User: arn:aws:sts::xxxxx:assumed-role/xxxx-ack-controller/xxxxx
      is not authorized to perform: iam:PassRole on resource: arn:aws:iam::xxxxx:role/rds-monitoring-role
      because no identity-based policy allows the iam:PassRole action\n\tstatus code:
      403, request id: xxxx"

Could you help us understand why this behavior is occurring? We believed that the ACK object in the Kubernetes cluster was supposed to be the definitive source of truth and that it should not be affected by manual changes made in AWS.

Steps to reproduce

Update the performanceInsightsEnabled setting of a DB instance through the AWS Console, then check the corresponding DBInstance object in the Kubernetes cluster to see if the change is reflected.

Expected outcome

ACK Object in Kubernetes Cluster: Typically, the Kubernetes Custom Resource (CR) managed by ACK is considered the source of truth. The controller continuously reconciles the state of the resource in AWS with the desired state specified in the Kubernetes CR.
AWS Console Changes: When you manually modify a resource via the AWS Console, this change is usually detected by the ACK controller during its reconciliation loop. The controller should attempt to revert the AWS resource to the state defined in the Kubernetes CR.

Environment Prod

Kubernetes version 1.28
Using EKS (yes/no), if so version?
AWS service targeted (S3, RDS, etc.) RDS

aws-controllers-k8s / community

ACK RDS Controller: performanceInsightsEnabled manual changes had different behavior #2123