search

crossplane-contrib / provider-upjet-aws

Official AWS Provider for Crossplane by Upbound.
https://marketplace.upbound.io/providers/upbound/provider-aws
Apache License 2.0
137 stars 112 forks source link

[Bug]: Replicationgroup.elasticache.aws.upbound.io in async after aws provider upgrade from 1.3.1 to 1.6.0 #1370

Open fola-ooo opened 1 week ago

fola-ooo commented 1 week ago

Is there an existing issue for this?

Affected Resource(s)

ReplicationGroup.elasticache.aws.upbound.io/v1beta2

Resource MRs required to reproduce the bug

apiVersion: elasticache.aws.upbound.io/v1beta2
kind: ReplicationGroup
metadata:
  name: ***-app
 spec:
  deletionPolicy: Delete
  forProvider:
    autoMinorVersionUpgrade: "true"
    automaticFailoverEnabled: false
    description: Elasticache-ReplicationGroup
    engine: redis
    engineVersion: "7.1"
    ipDiscovery: ipv4
    maintenanceWindow: mon:08:00-mon:11:00
    multiAzEnabled: false
    networkType: ipv4
    nodeType: cache.t2.micro
    numCacheClusters: 1
    parameterGroupName: default.redis7
    port: 6379
    region: eu-central-1
    replicasPerNodeGroup: 0
    snapshotWindow: 03:00-04:00
    subnetGroupName: ***-app
    subnetGroupNameRef:
      name: ***-app
    subnetGroupNameSelector:
      matchControllerRef: true
  providerConfigRef:
    name: provider-aws
  writeConnectionSecretToRef:
    name: ***-app-elasticache-connection
    namespace: default

Steps to Reproduce

Using the manifest above, create replication group with all upbound providers and aws family in version 1.3.1. Upgrade elasticache provider to 1.6.0

What happened?

Replication groups went into Async state

Relevant Error Output Snippet

conditions:
  - lastTransitionTime: "2024-06-18T14:34:01Z"
    message: "update failed: async update failed: failed to update the resource: [{0
      modifying ElastiCache Replication Group (***-app) authentication: InvalidParameterValue:
      Invalid AUTH token provided. Please check valid AUTH token format.\n\tstatus
      code: 400, request id: fdf564b9-8fee-4ada-b54d-737dc0bbb738  []}]"
    reason: ReconcileError
    status: "False"
    type: Synced
  - lastTransitionTime: "2024-06-18T14:30:30Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2024-06-18T14:34:01Z"
    message: "async update failed: failed to update the resource: [{0 modifying ElastiCache
      Replication Group (***-app) authentication: InvalidParameterValue: Invalid
      AUTH token provided. Please check valid AUTH token format.\n\tstatus code: 400,
      request id: fdf564b9-8fee-4ada-b54d-737dc0bbb738  []}]"
    reason: AsyncUpdateFailure
    status: "False"
    type: LastAsyncOperation

Crossplane Version

1.15.3

Provider Version

1.6.0

Kubernetes Version

1.29.2

Kubernetes Distribution

EKS

Additional Info

No response

caiofralmeida commented 1 week ago

@fola-ooo At version v1.6.1 there is a new field autoGenerateAuthToken, maybe trying to force as false, solves this issue.

dbs-gong commented 1 week ago

this issue is also apperent in on 1.7.0 it is no longer possible to configure a redis-elasticache cluster if usergroups are assigned and authtoken is disabled , after the creation process the replicationgroup will go out of sync with the fallowing error , 

Warning  CannotUpdateExternalResource  2m13s (x46 over 41m)  managed/elasticache.aws.upbound.io/v1beta2, kind=replicationgroup  (combined from similar events): async update failed: failed to update the resource: [{0 modifying ElastiCache Replication Group (new-test-v2) authentication: InvalidParameterCombination: Auth tokens can't be enabled with a user group already associated. Pass RemoveUserGroups to proceed.
           status code: 400, request id: ZZZZ

i have tested all combinations to allow this object to be synced and ready , here is my ForProvider config :

For Provider:
    Apply Immediately:           false
    At Rest Encryption Enabled:  true
    Auth Token Secret Ref:
      Key:
      Name:
      Namespace:
    Auto Generate Auth Token:    false
    Auto Minor Version Upgrade:  true
    Automatic Failover Enabled:  true
    Description:                 new-test-v2
    Engine:                      redis
    Engine Version:              7.1
    Ip Discovery:                ipv4
    Maintenance Window:          sun:05:00-sun:09:00
    Network Type:                ipv4
    Node Type:                   cache.t4g.micro
    Num Node Groups:             3
    Parameter Group Name:        new-test-v2-parameter-group
    Port:                        6379
    Region:                      eu-west-1
    Replicas Per Node Group:     1
    Security Group Id Refs:
      Name:  <REMOVED>
    Security Group Id Selector:
      Match Controller Ref:  true
    Security Group Ids:
      <REMOVED>
    Snapshot Retention Limit:  0
    Snapshot Window:           00:00-01:00
    Subnet Group Name:         new-test-v2-subnet-group
    Subnet Group Name Ref:
      Name:  new-test-v2-subnet-group
    Subnet Group Name Selector:
      Match Controller Ref:  true
    Tags:
      Crossplane - Kind:            replicationgroup.elasticache.aws.upbound.io
      Crossplane - Name:            new-test-v2
      Crossplane - Providerconfig:  default
    Transit Encryption Enabled:     true
    Transit Encryption Mode:        required
    User Group Ids:
      new-test-v2-user-group
  Init Provider:
    User Group Ids:
      new-test-v2-user-group
  Management Policies:
    *
  Provider Config Ref:
    Name:  default
Status:
  At Provider:
    Apply Immediately:               false
    Arn:                             <REMOVED>
    At Rest Encryption Enabled:      true
    Auto Minor Version Upgrade:      true
    Automatic Failover Enabled:      true
    Cluster Enabled:                 true
    Configuration Endpoint Address:  <REMOVED>
    Data Tiering Enabled:            false
    Description:                     new-test-v2
    Engine:                          redis
    Engine Version:                  7.1
    Engine Version Actual:           7.1.0
    Id:                              new-test-v2
    Ip Discovery:                    ipv4
    Kms Key Id:
    Maintenance Window:              sun:05:00-sun:09:00
    Member Clusters:
      new-test-v2-0001-001
      new-test-v2-0001-002
      new-test-v2-0002-001
      new-test-v2-0002-002
      new-test-v2-0003-001
      new-test-v2-0003-002
    Multi Az Enabled:         false
    Network Type:             ipv4
    Node Type:                cache.t4g.micro
    Num Cache Clusters:       6
    Num Node Groups:          3
    Parameter Group Name:     new-test-v2-parameter-group
    Port:                     6379
    Replicas Per Node Group:  1
    Security Group Ids:
      <REMOVED>
    Snapshot Retention Limit:  0
    Snapshot Window:           00:00-01:00
    Subnet Group Name:         new-test-v2-subnet-group
    Tags:
      Crossplane - Kind:            replicationgroup.elasticache.aws.upbound.io
      Crossplane - Name:            new-test-v2
      Crossplane - Providerconfig:  default
    Tags All:
      Crossplane - Kind:            replicationgroup.elasticache.aws.upbound.io
      Crossplane - Name:            new-test-v2
      Crossplane - Providerconfig:  default
    Transit Encryption Enabled:     true
    Transit Encryption Mode:        required
    User Group Ids:
      new-test-v2-user-group
  Conditions:
    Last Transition Time:  2024-06-23T13:15:26Z
    Reason:                Available
    Status:                True
    Type:                  Ready
    Last Transition Time:  2024-06-23T15:17:53Z
    Message:               update failed: async update failed: failed to update the resource: [{0 modifying ElastiCache Replication Group (new-test-v2) authentication: InvalidParameterCombination: Auth tokens can't be enabled with a user group already associated. Pass RemoveUserGroups to proceed.
                           status code: 400, request id: <REMOVED>  []}]
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
    Last Transition Time:  2024-06-23T15:17:53Z
    Message:               async update failed: failed to update the resource: [{0 modifying ElastiCache Replication Group (new-test-v2) authentication: InvalidParameterCombination: Auth tokens can't be enabled with a user group already associated. Pass RemoveUserGroups to proceed.

tried -

  1. not passing autoGenerateAuthToken
  2. passing autoGenerateAuthToken:false
  3. passing empty reference to Auth Token Secret Ref
  4. not passing any auth parameters .
  5. passing the userGroupId's only at initProvider and not in forProvider

all these attempts resulted with the same scenario - which after creation of replicationgroup fails the reconcile step . this is critical since in this state you cannot change any other setting of replicationgroup - like scaling - or upgrading version

please help