search

crossplane-contrib / provider-aws

Crossplane AWS Provider
Apache License 2.0
437 stars 372 forks source link

cannot modify RDS instance: api error InvalidParameterCombination: No modifications were requested #1795

Closed mateusz-lubanski-sinch closed 1 year ago

mateusz-lubanski-sinch commented 1 year ago

Our RDSInstance get into an SYNCED=False status with below event:

Warning  CannotUpdateExternalResource  4m56s (x2874 over 2d)  managed/rdsinstance.database.aws.crossplane.io  cannot modify RDS instance: api error InvalidParameterCombination: No modifications were requested

when I look into AWS CloudTrail I found request done by crossplane:

    "eventTime": "2023-06-29T10:59:50Z",
    "eventSource": "rds.amazonaws.com",
    "eventName": "ModifyDBInstance",
    "awsRegion": "eu-west-1",
    "userAgent": "aws-sdk-go-v2/1.16.16 os/linux lang/go/1.18.10 md/GOOS/linux md/GOARCH/amd64 api/rds/1.14.0 crossplane-provider-aws/v0.40.0",
    "errorCode": "InvalidParameterCombinationException",
    "errorMessage": "No modifications were requested",
    "requestParameters": {
        "dBInstanceIdentifier": "wac-ekswac001-rdswac002",
        "applyImmediately": false,
        "allowMajorVersionUpgrade": false,
        "optionGroupName": "sinch-default-mysql-5-7"
    },
    "responseElements": null,
    "requestID": "2a19f3ba-4dad-41b2-aa9b-5249bb1ed795",
    "eventID": "e08c9078-b07f-4078-b047-c49fb949308f",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,

optionG

When I set applyImmedietely to true it changes status to True but no changes are performed. This is on our production environment so we don't want set by default applyImmedietely to true

After setting applyImmedietely back to false I am getting again same Warning event

Crossplane version:

MisterMX commented 1 year ago

@mateusz-lubanski-sinch can you provide a sample MR to reproduce this issue?

mateusz-lubanski-sinch commented 1 year ago

Hey @MisterMX

Below is example of MR which is in SYNC=False state:

apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  annotations:
    crossplane.io/composition-resource-name: rds
    crossplane.io/external-create-failed: "2023-06-15T11:27:19Z"
    crossplane.io/external-create-succeeded: "2023-06-15T11:27:22Z"
    crossplane.io/external-name: wac-ekswac001-rdswac002
  creationTimestamp: "2023-06-15T11:27:17Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generateName: wac-ekswac001-rdswac002-
  generation: 11
  labels:
    crossplane.io/claim-name: ekswac001-rdswac002
    crossplane.io/claim-namespace: wac
    crossplane.io/composite: ekswac001-rdswac002-4tftn
  name: wac-ekswac001-rdswac002-8n7rg
  ownerReferences:
  - apiVersion: rds.aws.crossplane.sinch.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: XMySQLInstance
    name: ekswac001-rdswac002-4tftn
    uid: b44f14dc-04ae-4fb0-92f6-12043ff96eda
  resourceVersion: "369081543"
  uid: f9a3d6d0-2bb5-4e65-a5d5-91f4f3742271
spec:
  deletionPolicy: Orphan
  forProvider:
    allocatedStorage: 333
    applyModificationsImmediately: false
    autoMinorVersionUpgrade: true
    availabilityZone: eu-west-1a
    backupRetentionPeriod: 1
    caCertificateIdentifier: rds-ca-2019
    copyTagsToSnapshot: true
    dbInstanceClass: db.r5.2xlarge
    dbParameterGroupName: wac-ekswac001-rdswac002
    dbSubnetGroupName: private_subnets_vpc-***
    deletionProtection: false
    enableCloudwatchLogsExports:
    - audit
    enableIAMDatabaseAuthentication: false
    enablePerformanceInsights: true
    engine: mysql
    engineVersion: 5.7.38
    kmsKeyId: ***
    licenseModel: general-public-license
    masterUsername: root
    maxAllocatedStorage: 1000
    multiAZ: true
    optionGroupName: sinch-default-mysql-5-7
    performanceInsightsKMSKeyId: arn:aws:kms:eu-west-1:170628875025:key/***
    performanceInsightsRetentionPeriod: 7
    port: 3306
    preferredBackupWindow: 14:00-16:00
    preferredMaintenanceWindow: mon:02:30-mon:03:00
    publiclyAccessible: false
    region: eu-west-1
    skipFinalSnapshotBeforeDeletion: true
    storageEncrypted: true
    storageType: gp2
    tags:
    - key: OpsAutomator-Snap-Daily
      value: "true"
    - key: cg_env
      value: production
    - key: cg_iac
      value: crossplane
    - key: crossplane-kind
      value: rdsinstance.database.aws.crossplane.io
    - key: crossplane-name
      value: wac-ekswac001-rdswac002-8n7rg
    - key: crossplane-providerconfig
      value: crossplane-provider-aws
    vpcSecurityGroupIds:
    - sg-***
    - sg-***
  providerConfigRef:
    name: crossplane-provider-aws
  publishConnectionDetailsTo:
    configRef:
      name: default
    metadata:
      labels:
        secret.crossplane.io/owner-uid: f9a3d6d0-2bb5-4e65-a5d5-91f4f3742271
    name: wac-ekswac001-rdswac002-rdsinstance
status:
  atProvider:
    allocatedStorage: 333
    backupRetentionPeriod: 1
    dbInstanceArn: arn:aws:rds:eu-west-1:***:db:wac-ekswac001-rdswac002
    dbInstanceStatus: available
    dbParameterGroups:
    - dbParameterGroupName: wac-ekswac001-rdswac002
      parameterApplyStatus: in-sync
    dbResourceId: db-D4FA2UYRDIGMHTBGHX2K5KU2BQ
    dbSubnetGroup:
      dbSubnetGroupDescription: This is a private subnet group for AWS RDS instances.
        Only containing our regular private subnets in vpc-***.
      dbSubnetGroupName: private_subnets_vpc-***
      subnetGroupStatus: Complete
      subnets:
      - subnetAvailabilityZone:
          name: eu-west-1b
        subnetIdentifier: subnet-***
        subnetStatus: Active
      - subnetAvailabilityZone:
          name: eu-west-1a
        subnetIdentifier: subnet-***
        subnetStatus: Active
      - subnetAvailabilityZone:
          name: eu-west-1c
        subnetIdentifier: subnet-***
        subnetStatus: Active
      vpcId: vpc-7362b516
    endpoint:
      address: wac-ekswac001-rdswac002.***.eu-west-1.rds.amazonaws.com
      hostedZoneId: ***
      port: 3306
    instanceCreateTime: "2023-06-15T11:30:59Z"
    latestRestorableTime: "2023-07-04T05:30:00Z"
    optionGroupMemberships:
    - optionGroupName: sinch-default-mysql-5-7
      status: in-sync
    pendingModifiedValues:
      pendingCloudwatchLogsExports: {}
    performanceInsightsEnabled: true
    secondaryAvailabilityZone: eu-west-1c
    vpcSecurityGroups:
    - status: active
      vpcSecurityGroupId: sg-***
    - status: active
      vpcSecurityGroupId: sg-***
  conditions:
  - lastTransitionTime: "2023-06-15T11:32:53Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2023-07-01T21:53:33Z"
    message: 'update failed: cannot modify RDS instance: api error InvalidParameterCombination:
      No modifications were requested'
    reason: ReconcileError
    status: "False"
    type: Synced
wotolom commented 1 year ago

We are seeing a similar behavior with the field cloudwatchLogsExportConfiguration.

here the request from Cloudtrail:

"errorCode": "InvalidParameterCombinationException",
    "errorMessage": "No modifications were requested",
    "requestParameters": {
        "dBInstanceIdentifier": "baum-dev-rdsinstance2",
        "applyImmediately": false,
        "allowMajorVersionUpgrade": true,
        "cloudwatchLogsExportConfiguration": {
            "disableLogTypes": [
                "upgrade",
                "postgresql"
            ]
        }
    },

As already mentioned in the issue, changing applyImmediately to true results in no Error, however crossplane is constantly sending modify-requests, like this one:

"requestParameters": {
        "dBInstanceIdentifier": "baum-dev-rdsinstance",
        "applyImmediately": true,
        "allowMajorVersionUpgrade": true,
        "cloudwatchLogsExportConfiguration": {
            "disableLogTypes": [
                "upgrade",
                "postgresql"
            ]
        }
    },
    "responseElements": {
        "dBInstanceIdentifier": "baum-dev-rdsinstance",

also visible in the MR events:

Normal   UpdatedExternalResource          8m48s (x29 over 65m)  managed/rdsinstance.database.aws.crossplane.io  Successfully requested update of external resource

with the bit debugging I have done in the controller so far, my evaluation: the controller seems to be missing custom logic for handling the fields cloudwatchLogsExportConfiguration and also optionGroupName in isUpToDate(). There are no direct matching fields/parameters coming from AWS through DescribeDBInstances , therefore the controller always sees a diff.

OptionGroupName probably needs to be checked against OptionGroupMemberships For CloudwatchLogsExportConfiguration, I'm unsure. Maybe EnabledCloudwatchLogsExports. But here there is an identical field named EnableCloudwatchLogsExports in RDSInstanceParameters.

I probably will try to fix this in the next days.

mateusz-lubanski-sinch commented 1 year ago

Hey @wotolom & @MisterMX Can we please re-open this issue? After upgrading provider-aws to v0.42.0 problem seems still occur

current provider

kubectl get providers.pkg.crossplane.io | grep 'crossplane-contrib/provider-aws'
NAME                                  INSTALLED   HEALTHY   PACKAGE                                                         AGE
crossplane-provider-aws               True        True      xpkg.upbound.io/crossplane-contrib/provider-aws:v0.42.0         201d

ManagedResource state:

kubectl get rdsinstances.database.aws.crossplane.io wac-ekswac001-rdswac001-h58tx
NAME                            READY   SYNCED   STATE       ENGINE   VERSION   AGE
wac-ekswac001-rdswac001-h58tx   True    False    available   mysql    5.7.38    158d

kubectl describe rdsinstances.database.aws.crossplane.io wac-ekswac001-rdswac001-h58tx
...
  Conditions:
    Last Transition Time:  2023-07-18T04:28:07Z
    Message:               update failed: cannot modify RDS instance: api error InvalidParameterCombination: No modifications were requested
    Reason:                ReconcileError
    Status:                False
    Type:                  Synced
    Last Transition Time:  2023-06-02T06:06:53Z
    Reason:                Available
    Status:                True
    Type:                  Ready
Events:
  Type     Reason                        Age                      From                                            Message
  ----     ------                        ----                     ----                                            -------
  Warning  CannotUpdateExternalResource  2m51s (x580 over 3h18m)  managed/rdsinstance.database.aws.crossplane.io  cannot modify RDS instance: api error InvalidParameterCombination: No modifications were requested

AWS CloudTrail Log:

CloudTrail Log
    "errorCode": "InvalidParameterCombinationException",
    "errorMessage": "No modifications were requested",
    "requestParameters": {
        "dBInstanceIdentifier": "us1-ekswac001-rdswac001",
        "applyImmediately": false,
        "allowMajorVersionUpgrade": false,
        "optionGroupName": "sinch-default-mysql-5-7"
    },

crossplane version v1.13.2

mateusz-lubanski-sinch commented 1 year ago

ahh I see, thanks a lot for that fix! <3

wotolom commented 1 year ago

Hey @mateusz-lubanski-sinch,

sorry for my previous (wrong and now deleted) comment, that fix coming with provider-aws v0.43.0 is similar, but applies to another field (CloudwatchLogsExportConfiguration / EnableCloudwatchLogsExports).

You are actually right with expecting the fix for optionGroupName being already included in provider-aws v0.42.0!

I just tested this, but for me it seems to work fine in provider-aws v0.42.0:

Status:
  At Provider:
...
 Option Group Memberships:
      Option Group Name:  custom-default-mysql-8
      Status:             in-sync
...
  Conditions:
    Last Transition Time:  2023-08-29T12:13:23Z
    Reason:                Available
    Status:                True
    Type:                  Ready
    Last Transition Time:  2023-08-29T12:29:52Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced
Events:
  Type     Reason                           Age                From                                            Message
  ----     ------                           ----               ----                                            -------
  Normal   CreatedExternalResource          45m                managed/rdsinstance.database.aws.crossplane.io  Successfully requested creation of external resource

Also the case where the optionGroupName is actually changed in the claim while applyModificationsImmediately is false, I do not get the update-loop:

Status:
  At Provider:
...
Option Group Memberships:
      Option Group Name:  default:mysql-5-7
      Status:             pending-maintenance-apply
      Option Group Name:  sinch-default-mysql-5-7
      Status:             pending-maintenance-removal
...
  Conditions:
    Last Transition Time:  2023-08-29T12:34:21Z
    Reason:                Available
    Status:                True
    Type:                  Ready
    Last Transition Time:  2023-08-29T12:30:19Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced
Events:
  Type    Reason                   Age    From                                            Message
  ----    ------                   ----   ----                                            -------
  Normal  CreatedExternalResource  30m    managed/rdsinstance.database.aws.crossplane.io  Successfully requested creation of external resource
  Normal  UpdatedExternalResource  9m26s  managed/rdsinstance.database.aws.crossplane.io  Successfully requested update of external resource
mateusz-lubanski-sinch commented 1 year ago

I can share my RDSInstance ManagedResource if this helps:

apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  annotations:
    crossplane.io/composition-resource-name: rds
    crossplane.io/external-name: us1-ekswac001-rdswac001    
  creationTimestamp: "2023-03-23T14:43:16Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generateName: wac-ekswac001-rdswac001-
  generation: 129675
  labels:
    crossplane.io/claim-name: ekswac001-rdswac001
    crossplane.io/claim-namespace: wac
    crossplane.io/composite: ekswac001-rdswac001-4tshm
  name: wac-ekswac001-rdswac001-h58tx
  ownerReferences:
  - apiVersion: rds.aws.crossplane.sinch.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: XMySQLInstance
    name: ekswac001-rdswac001-4tshm
    uid: a8211232-ea1e-429e-b477-ebfbecc20058
  resourceVersion: "446399353"
  uid: 05a94c62-ddaf-4ee7-b885-a3704f782091
spec:
  deletionPolicy: Orphan
  forProvider:
    allocatedStorage: 333
    applyModificationsImmediately: false
    autoMinorVersionUpgrade: true
    availabilityZone: us-east-1a
    backupRetentionPeriod: 1
    caCertificateIdentifier: rds-ca-2019
    copyTagsToSnapshot: true
    dbInstanceClass: db.r5.4xlarge
    dbName: db
    dbParameterGroupName: rdswac001-db-parameter-group
    dbSubnetGroupName: private_subnets_vpc-xxx
    deletionProtection: false
    enableCloudwatchLogsExports:
    - audit
    enableIAMDatabaseAuthentication: false
    enablePerformanceInsights: true
    engine: mysql
    engineVersion: 5.7.38
    kmsKeyId: 6a1a9e12-xxx-xxx-xxx-xxx
    licenseModel: general-public-license
    masterPasswordSecretRef:
      key: password
      name: ekswac001-rdswac001-master-password
      namespace: wac
    masterUsername: root
    maxAllocatedStorage: 1000
    monitoringInterval: 60
    monitoringRoleArn: arn:aws:iam::xxx:role/rds-monitoring-role
    multiAZ: true
    optionGroupName: sinch-default-mysql-5-7
    performanceInsightsKMSKeyId: arn:aws:kms:us-east-1:xxx:key/6a1a9e12-xxx-xxx-xxx-xxx
    performanceInsightsRetentionPeriod: 7
    port: 3306
    preferredBackupWindow: 14:00-16:00
    preferredMaintenanceWindow: mon:02:30-mon:03:00
    publiclyAccessible: false
    region: us-east-1
    skipFinalSnapshotBeforeDeletion: true
    storageEncrypted: true
    storageType: gp2
    tags:
    - key: crossplane-kind
      value: rdsinstance.database.aws.crossplane.io
    - key: crossplane-name
      value: wac-ekswac001-rdswac001-h58tx
    - key: crossplane-providerconfig
      value: crossplane-provider-aws
    vpcSecurityGroupIds:
    - sg-0fxxx
    - sg-09xxx
  providerConfigRef:
    name: crossplane-provider-aws
  publishConnectionDetailsTo:
    configRef:
      name: default
    metadata:
      labels:
        secret.crossplane.io/owner-uid: 05a94c62-ddaf-4ee7-b885-a3704f782091
    name: wac-ekswac001-rdswac001-rdsinstance
status:
  atProvider:
    allocatedStorage: 333
    backupRetentionPeriod: 1
    dbInstanceArn: arn:aws:rds:us-east-1:xxx:db:us1-ekswac001-rdswac001
    dbInstanceStatus: available
    dbParameterGroups:
    - dbParameterGroupName: rdswac001-db-parameter-group
      parameterApplyStatus: in-sync
    dbResourceId: db-xxx
    dbSubnetGroup:
      dbSubnetGroupDescription: This is a private subnet group for AWS RDS instances.
        Only containing our regular private subnets in vpc-xxx
      dbSubnetGroupName: private_subnets_vpc-xxx
      subnetGroupStatus: Complete
      subnets:
      - subnetAvailabilityZone:
          name: us-east-1c
        subnetIdentifier: subnet-5xxx
        subnetStatus: Active
      - subnetAvailabilityZone:
          name: us-east-1b
        subnetIdentifier: subnet-0xxx
        subnetStatus: Active
      - subnetAvailabilityZone:
          name: us-east-1a
        subnetIdentifier: subnet-exxx
        subnetStatus: Active
      vpcId: vpc-xxx
    endpoint:
      address: us1-ekswac001-rdswac001.xxx.us-east-1.rds.amazonaws.com
      hostedZoneId: Z2R2ITUGPM61AM
      port: 3306
    enhancedMonitoringResourceArn: arn:aws:logs:us-east-1:xxx:log-group:RDSOSMetrics:log-stream:db-xxx
    instanceCreateTime: "2022-08-04T06:34:53Z"
    latestRestorableTime: "2023-08-30T11:00:00Z"
    optionGroupMemberships:
    - optionGroupName: sinch-default-mysql-5-7
      status: in-sync
    pendingModifiedValues:
      pendingCloudwatchLogsExports: {}
    performanceInsightsEnabled: true
    secondaryAvailabilityZone: us-east-1c
    vpcSecurityGroups:
    - status: active
      vpcSecurityGroupId: sg-0fxxx
    - status: active
      vpcSecurityGroupId: sg-09xxx
  conditions:
  - lastTransitionTime: "2023-07-18T04:28:07Z"
    message: 'update failed: cannot modify RDS instance: api error InvalidParameterCombination:
      No modifications were requested'
    reason: ReconcileError
    status: "False"
    type: Synced
  - lastTransitionTime: "2023-06-02T06:06:53Z"
    reason: Available
    status: "True"
    type: Ready

Above ManagedResource was not changed before or after provider-aws upgrade to 0.42.0

wotolom commented 1 year ago

Thanks that helped!

The kmsKeyId field value is triggering the update because the controller compares it with the ARN of the kmsKey, which is what AWS provides - regardless of the way the kmsKey was set (ID, ARN, alias...) . However the kmsKeyId is actually immutable and is not part of the ModifyInput. Therefore it was not listed in the CloudTrail logs. The reason we still saw the optionGroupName is, because it is still part of the patch the controller creates for the ModifyInput, even though it is upToDate-checked independently and not the reason for the update-trigger.

I will provide a small fix that ignored kmsKeyId in isUpToDate() and probably include optionGroupName in lateInit() to avoid it appearing in the patch.

@mateusz-lubanski-sinch if you can change the kmsKeyId value to the ARN of the kmsKey (like in performanceInsightsKMSKeyId), the update calls should stop - if my analysis is complete and there are no other wrongly checked fields

mateusz-lubanski-sinch commented 1 year ago

I updated kmsKey (replaced id with arn) but still in events I can see:

kubectl get rdsinstances.database.aws.crossplane.io wac-ekswac001-rdswac001-h58tx -o yaml | grep "kms"
    kmsKeyId: arn:aws:kms:us-east-1:xxxx:key/6a1a9e12-xxxx-xxxx-xxxx-xxxx
    performanceInsightsKMSKeyId: arn:aws:kms:us-east-1:xxxx:key/6a1a9e12-xxxx-xxxx-xxxx-xxxx

kubectl describe rdsinstances.database.aws.crossplane.io wac-ekswac001-rdswac001-h58tx
...
Events:
  Type     Reason                        Age                        From                                            Message
  ----     ------                        ----                       ----                                            -------
  Normal   UpdatedExternalResource       8m11s                      managed/rdsinstance.database.aws.crossplane.io  Successfully requested update of external resource
  Warning  CannotUpdateExternalResource  3m10s (x20063 over 4d18h)  managed/rdsinstance.database.aws.crossplane.io  cannot modify RDS instance: api error InvalidParameterCombination: No modifications were requested
wotolom commented 1 year ago

Could you please check, if the behavior stays identical after you upgrade to provider-aws v0.43.0?

If it does, please look into the provider-aws (pod) logs and search for logs concerning the specific rdsinstance. Try to find the log message that includes the observed diff. (it contains the following string: https://github.com/crossplane-contrib/provider-aws/blob/d5f66752e7fdcc2097040f749f8cc5018260bb09/pkg/clients/database/rds.go#L725)

mateusz-lubanski-sinch commented 1 year ago

I am still getting same event with 0.43.0

kubectl --context us1-ekswac001 describe rdsinstances.database.aws.crossplane.io wac-ekswac001-rdswac001-h58tx
...
Events:
  Type     Reason                        Age                      From                                            Message
  ----     ------                        ----                     ----                                            -------
  Warning  CannotUpdateExternalResource  15m (x20467 over 4d21h)  managed/rdsinstance.database.aws.crossplane.io  cannot modify RDS instance: api error InvalidParameterCombination: No modifications were requested
  Warning  CannotUpdateExternalResource  6m8s (x20 over 10m)      managed/rdsinstance.database.aws.crossplane.io  cannot modify RDS instance: api error InvalidParameterCombination: No modifications were requested
  Warning  CannotUpdateExternalResource  12s (x22 over 5m31s)     managed/rdsinstance.database.aws.crossplane.io  cannot modify RDS instance: api error InvalidParameterCombination: No modifications were requested

Should I enable debug logs to see that message?

kubectl logs -n crossplane-system pod/crossplane-provider-aws-447dacb385f5-d6f4645df-cfsq6

{"level":"info","ts":"2023-09-04T09:48:19.137Z","logger":"provider-aws","msg":"Alpha feature enabled","flag":"EnableAlphaExternalSecretStores"}
Setup endpointgroup
W0904 09:48:19.293481       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.Listener: listeners.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "listeners" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
E0904 09:48:19.293517       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.Listener: failed to list *v1alpha1.Listener: listeners.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "listeners" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
W0904 09:48:19.302218       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.Accelerator: accelerators.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "accelerators" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
E0904 09:48:19.302247       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.Accelerator: failed to list *v1alpha1.Accelerator: accelerators.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "accelerators" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
W0904 09:48:19.304195       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.EndpointGroup: endpointgroups.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "endpointgroups" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
E0904 09:48:19.304209       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.EndpointGroup: failed to list *v1alpha1.EndpointGroup: endpointgroups.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "endpointgroups" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
W0904 09:48:19.998812       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.AccessPoint: accesspoints.s3control.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "accesspoints" in API group "s3control.aws.crossplane.io" at the cluster scope
E0904 09:48:19.998838       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.AccessPoint: failed to list *v1alpha1.AccessPoint: accesspoints.s3control.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "accesspoints" in API group "s3control.aws.crossplane.io" at the cluster scope
W0904 09:48:20.286015       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.Listener: listeners.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "listeners" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
E0904 09:48:20.286038       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.Listener: failed to list *v1alpha1.Listener: listeners.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "listeners" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
W0904 09:48:20.543295       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.Accelerator: accelerators.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "accelerators" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
E0904 09:48:20.543322       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.Accelerator: failed to list *v1alpha1.Accelerator: accelerators.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "accelerators" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
W0904 09:48:20.604803       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.EndpointGroup: endpointgroups.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "endpointgroups" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
E0904 09:48:20.604832       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.EndpointGroup: failed to list *v1alpha1.EndpointGroup: endpointgroups.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "endpointgroups" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
W0904 09:48:21.437828       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.AccessPoint: accesspoints.s3control.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "accesspoints" in API group "s3control.aws.crossplane.io" at the cluster scope
E0904 09:48:21.437857       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.AccessPoint: failed to list *v1alpha1.AccessPoint: accesspoints.s3control.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "accesspoints" in API group "s3control.aws.crossplane.io" at the cluster scope
W0904 09:48:22.246244       1 warnings.go:70] BucketPolicy has been deprecated. Use spec.forProvider.policy in Bucket instead.
W0904 09:48:22.247345       1 warnings.go:70] BucketPolicy has been deprecated. Use spec.forProvider.policy in Bucket instead.
W0904 09:48:22.293394       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.Accelerator: accelerators.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "accelerators" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
E0904 09:48:22.293417       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.Accelerator: failed to list *v1alpha1.Accelerator: accelerators.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "accelerators" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
W0904 09:48:23.005988       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.EndpointGroup: endpointgroups.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "endpointgroups" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
E0904 09:48:23.006027       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.EndpointGroup: failed to list *v1alpha1.EndpointGroup: endpointgroups.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "endpointgroups" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
W0904 09:48:23.033909       1 reflector.go:533] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: failed to list *v1alpha1.Listener: listeners.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "listeners" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
E0904 09:48:23.033938       1 reflector.go:148] k8s.io/client-go@v0.27.3/tools/cache/reflector.go:231: Failed to watch *v1alpha1.Listener: failed to list *v1alpha1.Listener: listeners.globalaccelerator.aws.crossplane.io is forbidden: User "system:serviceaccount:crossplane-system:crossplane-provider-aws-447dacb385f5" cannot list resource "listeners" in API group "globalaccelerator.aws.crossplane.io" at the cluster scope
[controller-runtime] log.SetLogger(...) was never called, logs will not be displayed:
goroutine 3530 [running]:
runtime/debug.Stack()
        runtime/debug/stack.go:24 +0x65
sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
        sigs.k8s.io/controller-runtime@v0.15.0/pkg/log/log.go:59 +0xbd
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).WithValues(0xc0008dd6c0, {0xc00256b360, 0x2, 0x2})
        sigs.k8s.io/controller-runtime@v0.15.0/pkg/log/deleg.go:168 +0x54
github.com/go-logr/logr.Logger.WithValues(...)
        github.com/go-logr/logr@v1.2.4/logr.go:323
sigs.k8s.io/controller-runtime/pkg/builder.(*Builder).doController.func1(0xc00256b340)
        sigs.k8s.io/controller-runtime@v0.15.0/pkg/builder/controller.go:398 +0x182
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0008ad860, {0x749d9a0, 0xc0000cfec0}, {0x625de60?, 0xc00260b880?})
        sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:305 +0x1b9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0008ad860, {0x749d9a0, 0xc0000cfec0})
        sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
        sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:222 +0x333
W0904 09:55:06.249893       1 warnings.go:70] BucketPolicy has been deprecated. Use spec.forProvider.policy in Bucket instead.
mateusz-lubanski-sinch commented 1 year ago

@wotolom should I create new issue or should we re-open this one?

wotolom commented 1 year ago

I think it is best to create a new issue, because for me it looks like same error message but different underlying error source.