[Bug]: Account MR unable to reconcile after rotating primary access key

[jaylevin]

Is there an existing issue for this?

• [X] I have searched the existing issues

Affected Resource(s)

storage.azure.upbound.io/v1beta1 - Account

Resource MRs required to reproduce the bug

apiVersion: storage.azure.upbound.io/v1beta1
kind: Account
metadata:
  name: test-account
spec:
  forProvider:
    accountKind: StorageV2
    accountReplicationType: GRS
    accountTier: Standard
    allowNestedItemsToBePublic: false
    blobProperties:
      - containerDeleteRetentionPolicy:
          - days: 30
        deleteRetentionPolicy:
          - days: 30
        versioningEnabled: true
    enableHttpsTrafficOnly: true
    location: eastus
    minTlsVersion: TLS1_2
    resourceGroupName: test-resource-group
  managementPolicies:
    - Create
    - Update
    - Observe
  providerConfigRef:
    name: test-dev-us-azure
  writeConnectionSecretToRef:
    name: redis-connection-secret
    namespace: test-dev-us

Steps to Reproduce

1. Deploy Account MR.
2. Login to Azure Portal, rotate the primary access key:
3. Wait for Azure Provider to reconcile the MR. The provider fails to observe the external resource and a 403 error is observed in the provider pod's logs. See error snippet below.

What happened?

Expected: The provider should reconcile the Storage Account's connection secret and update it with the new value that was generated via Azure Portal.

Observed: The Account MR becomes unsynced, and the azure-storage provider fails to observe the external resource due to 403 error below.

Relevant Error Output Snippet

2024-05-07T20:33:57Z    DEBUG   provider-azure  Cannot observe external resource    {"controller": "managed/storage.azure.upbound.io/v1beta1, kind=account", "request": {"name":"sap-cic-jrl-dev-redis-l54gs-5rnjg"}, "uid": "805c293c-10e6-40f3-b7f4-457e7b31f360", "version": "44012110", "external-name": "sapcicjrldevredis", "error": "failed to observe the resource: [{0 retrieving queue properties for Storage Account (Subscription: \"9c7ab030-74ff-4979-933b-34ca6ec76d3d\"\nResource Group Name: \"sap-cic-jrl-dev\"\nStorage Account Name: \"sapcicjrldevredis\"): queues.Client#GetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthenticationFailed\" Message=\"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\\nRequestId:c1c93d0e-6003-0090-0cbd-a02a7d000000\\nTime:2024-05-07T20:33:57.8899327Z\"  []}]", "errorVerbose": "failed to observe the resource: [{0 retrieving queue properties for Storage Account (Subscription: \"9c7ab030-74ff-4979-933b-34ca6ec76d3d\"\nResource Group Name: \"sap-cic-jrl-dev\"\nStorage Account Name: \"sapcicjrldevredis\"): queues.Client#GetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthenticationFailed\" Message=\"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\\nRequestId:c1c93d0e-6003-0090-0cbd-a02a7d000000\\nTime:2024-05-07T20:33:57.8899327Z\"  []}]\ngithub.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).Observe\n\tgithub.com/crossplane/upjet@v1.3.0-rc.0.0.20240319124750-50919febc5ab/pkg/controller/external_tfpluginsdk.go:471\ngithub.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Observe\n\tgithub.com/crossplane/upjet@v1.3.0-rc.0.0.20240319124750-50919febc5ab/pkg/controller/external_async_tfpluginsdk.go:126\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.15.1/pkg/reconciler/managed/reconciler.go:903\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.15.1/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1650"}

Crossplane Version

1.15.1

Provider Version

azure-family: v0.42.0, azure-storage: v1.0.1

Kubernetes Version

1.26

Kubernetes Distribution

No response

Additional Info

Restarting the azure-storage pod fixes the issue almost immediately and the MR becomes synced and ready. The connection secret redis-connection-secret is reconciled with the new credentials as expected.

[turkenf]

Hi @jaylevin,

Thank you for raising this issue. I can reproduce this issue in provider v1.1.0 with the information given.

    message: |-
      observe failed: failed to observe the resource: [{0 retrieving queue properties for Storage Account (Subscription: "038f2b7c-3265-43b8-8624-c9ad5da610a8"
      Resource Group Name: "fatihtestrg"
      Storage Account Name: "aluexample"): queues.Client#GetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthenticationFailed" Message="Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:4e08fd4c-6003-006b-309c-a3fa3e000000\nTime:2024-05-11T12:10:27.8970415Z"  []}]

[fortis-lt]

Hello, everyone Does anyone have any updates on this issue? I'm experiencing the same problem in version 1.16.0. In fact, Crossplane can't remove resources because of it.

[github-actions[bot]]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[fortis-lt]

/fresh

[fortis-lt]

Hello, folks Any updates for that case? I have similar problem and couldn't proper manage secrets rotation for crossplane resources