[Bug]: Account MR unable to reconcile after rotating primary access key

[jaylevin]

Is there an existing issue for this?

• [X] I have searched the existing issues

Affected Resource(s)

storage.azure.upbound.io/v1beta1 - Account

Resource MRs required to reproduce the bug

apiVersion: storage.azure.upbound.io/v1beta1
kind: Account
metadata:
  name: test-account
spec:
  forProvider:
    accountKind: StorageV2
    accountReplicationType: GRS
    accountTier: Standard
    allowNestedItemsToBePublic: false
    blobProperties:
      - containerDeleteRetentionPolicy:
          - days: 30
        deleteRetentionPolicy:
          - days: 30
        versioningEnabled: true
    enableHttpsTrafficOnly: true
    location: eastus
    minTlsVersion: TLS1_2
    resourceGroupName: test-resource-group
  managementPolicies:
    - Create
    - Update
    - Observe
  providerConfigRef:
    name: test-dev-us-azure
  writeConnectionSecretToRef:
    name: redis-connection-secret
    namespace: test-dev-us

Steps to Reproduce

1. Deploy Account MR.
2. Login to Azure Portal, rotate the primary access key:
3. Wait for Azure Provider to reconcile the MR. The provider fails to observe the external resource and a 403 error is observed in the provider pod's logs. See error snippet below.

What happened?

Expected: The provider should reconcile the Storage Account's connection secret and update it with the new value that was generated via Azure Portal.

Observed: The Account MR becomes unsynced, and the azure-storage provider fails to observe the external resource due to 403 error below.

Relevant Error Output Snippet

2024-05-07T20:33:57Z    DEBUG   provider-azure  Cannot observe external resource    {"controller": "managed/storage.azure.upbound.io/v1beta1, kind=account", "request": {"name":"sap-cic-jrl-dev-redis-l54gs-5rnjg"}, "uid": "805c293c-10e6-40f3-b7f4-457e7b31f360", "version": "44012110", "external-name": "sapcicjrldevredis", "error": "failed to observe the resource: [{0 retrieving queue properties for Storage Account (Subscription: \"9c7ab030-74ff-4979-933b-34ca6ec76d3d\"\nResource Group Name: \"sap-cic-jrl-dev\"\nStorage Account Name: \"sapcicjrldevredis\"): queues.Client#GetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthenticationFailed\" Message=\"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\\nRequestId:c1c93d0e-6003-0090-0cbd-a02a7d000000\\nTime:2024-05-07T20:33:57.8899327Z\"  []}]", "errorVerbose": "failed to observe the resource: [{0 retrieving queue properties for Storage Account (Subscription: \"9c7ab030-74ff-4979-933b-34ca6ec76d3d\"\nResource Group Name: \"sap-cic-jrl-dev\"\nStorage Account Name: \"sapcicjrldevredis\"): queues.Client#GetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"AuthenticationFailed\" Message=\"Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\\nRequestId:c1c93d0e-6003-0090-0cbd-a02a7d000000\\nTime:2024-05-07T20:33:57.8899327Z\"  []}]\ngithub.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKExternal).Observe\n\tgithub.com/crossplane/upjet@v1.3.0-rc.0.0.20240319124750-50919febc5ab/pkg/controller/external_tfpluginsdk.go:471\ngithub.com/crossplane/upjet/pkg/controller.(*terraformPluginSDKAsyncExternal).Observe\n\tgithub.com/crossplane/upjet@v1.3.0-rc.0.0.20240319124750-50919febc5ab/pkg/controller/external_async_tfpluginsdk.go:126\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.15.1/pkg/reconciler/managed/reconciler.go:903\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.15.1/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1650"}

Crossplane Version

1.15.1

Provider Version

azure-family: v0.42.0, azure-storage: v1.0.1

Kubernetes Version

1.26

Kubernetes Distribution

No response

Additional Info

Restarting the azure-storage pod fixes the issue almost immediately and the MR becomes synced and ready. The connection secret redis-connection-secret is reconciled with the new credentials as expected.

[turkenf]

Hi @jaylevin,

Thank you for raising this issue. I can reproduce this issue in provider v1.1.0 with the information given.

    message: |-
      observe failed: failed to observe the resource: [{0 retrieving queue properties for Storage Account (Subscription: "038f2b7c-3265-43b8-8624-c9ad5da610a8"
      Resource Group Name: "fatihtestrg"
      Storage Account Name: "aluexample"): queues.Client#GetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthenticationFailed" Message="Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:4e08fd4c-6003-006b-309c-a3fa3e000000\nTime:2024-05-11T12:10:27.8970415Z"  []}]