search

crossplane-contrib / provider-upjet-azure

Azure Provider for Crossplane.
https://marketplace.upbound.io/providers/upbound/provider-family-azure/
Apache License 2.0
61 stars 75 forks source link

[Bug]: No 'LateInitialize' management policy combined with an ungraceful provider pod termination can lead to attempted resource recreation #802

Open b-deam opened 2 months ago

b-deam commented 2 months ago

Is there an existing issue for this?

Affected Resource(s)

Resource MRs required to reproduce the bug

apiVersion: authorization.azure.upbound.io/v1beta1
kind: RoleAssignment
metadata:
  name: my-test
spec:
  deletionPolicy: Delete
  forProvider:
    name: <GUID>
    principalId: <PRINCIPALID>
    principalType: ServicePrincipal
    roleDefinitionName: Contributor
    scope: /subscriptions/<SUB_ID>
  initProvider: {}
  managementPolicies:
  - Create
  - Update
  - Delete
  - Observe
  providerConfigRef:
    name: azure-provider-config
---
apiVersion: managedidentity.azure.upbound.io/v1beta1
kind: UserAssignedIdentity
metadata:
  name: user-assigned-identity-test
spec:
  deletionPolicy: Delete
  forProvider:
    location: eastus2
    name: user-assigned-identity-test
    resourceGroupName: <RG_NAME>
  initProvider: {}
  managementPolicies:
  - Create
  - Update
  - Delete
  - Observe
  providerConfigRef:
    name: azure-provider-config

Steps to Reproduce

  1. Forcefully delete the respective provider pod(s), e.g. kubectl delete pod/provider-azure-authorization-ca718ab078b2-9c7fcdd55-hwrfn -n crossplane --force --grace-period=0, this may take a few tries
  2. Observe that the provider attempts to recreate the resource

What happened?

Resources should not be attempted to be created, rather if a field changes that requires resource recreation we should see an async update failure similar to:

NAME                                                      SYNCED   READY   STATUS

UserAssignedIdentity/my-user-assigned-id                             False    True    ReconcileError: update failed: async update
failed: refuse to update the external resource because the following update requires replacing it: cannot change the value of the argument "resource_group_name" from "app-1" to ""

Relevant Error Output Snippet

Crossplane beta trace for RoleAssignment:

NAME                     RESOURCE   SYNCED   READY   STATUS
RoleAssignment/my-test              False    False   ReconcileError: create failed: async create failed: failed to create the resource: [{0 A resource with the ID "/subscriptions/<subscription_id>/providers/Microsoft.Authorization/roleAssignments/11f7c29f-df3c-4b4f-99cb-a41e2f951bec" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_ro
le_assignment" for more information.  []}]

Logs from provider-azure-authorization after force deleting pod:

2024-08-28T04:35:11Z    DEBUG   provider-azure  Starting    {"sync-interval": "1h0m0s", "poll-interval": "10m0s", "poll-jitter": "30s", "max-reconcile-rate": 100}
2024-08-28T04:35:14Z    INFO    provider-azure  Beta feature enabled    {"flag": "EnableBetaManagementPolicies"}
2024-08-28T04:35:14Z    DEBUG   provider-azure  Calling the inner handler for Create event. {"gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "name": "my-test", "queueLength": 0}
2024-08-28T04:35:14Z    DEBUG   provider-azure  Reconciling {"controller": "managed/authorization.azure.upbound.io/v1beta1, kind=roleassignment", "request": {"name":"my-test"}}
2024-08-28T04:35:14Z    DEBUG   provider-azure  Connecting to the service provider  {"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment"}
2024-08-28T04:35:18Z    DEBUG   provider-azure  Instance state not found in cache, reconstructing...    {"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment"}
2024-08-28T04:35:18Z    DEBUG   provider-azure  Observing the external resource {"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment"}
2024-08-28T04:35:18Z    DEBUG   provider-azure  Diff detected   {"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"11f7c29f-df3c-4b4f-99cb-a41e2f951bec\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"principal_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"c2c23f94-189a-4a64-b46a-4eb52739ea49\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"principal_type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"ServicePrincipal\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"role_definition_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Contributor\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"scope\":*terraform.ResourceAttrDiff{Old:\"\", New:\"/subscriptions/<SUB_ID>\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"skip_service_principal_aad_check\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-08-28T04:35:18Z    DEBUG   provider-azure  Async create starting...    {"trackerUID": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "resourceName": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "tfID": ""}
2024-08-28T04:35:18Z    DEBUG   provider-azure  Creating the external resource  {"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment"}
2024-08-28T04:35:18Z    DEBUG   provider-azure  Successfully requested creation of external resource    {"controller": "managed/authorization.azure.upbound.io/v1beta1, kind=roleassignment", "request": {"name":"my-test"}, "uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "version": "572125", "external-name": "", "external-name": ""}
2024-08-28T04:35:18Z    DEBUG   provider-azure  Calling the inner handler for Update event. {"gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "name": "my-test", "queueLength": 0}
2024-08-28T04:35:18Z    DEBUG   provider-azure  Reconciling {"controller": "managed/authorization.azure.upbound.io/v1beta1, kind=roleassignment", "request": {"name":"my-test"}}
2024-08-28T04:35:18Z    DEBUG   provider-azure  Connecting to the service provider  {"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment"}
2024-08-28T04:35:19Z    DEBUG   provider-azure  Async create ended. {"trackerUID": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "resourceName": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "error": "async create failed: failed to create the resource: [{0 A resource with the ID \"/subscriptions/<SUB_ID>/providers/Microsoft.Authorization/roleAssignments/11f7c29f-df3c-4b4f-99cb-a41e2f951bec\" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for \"azurerm_role_assignment\" for more information.  []}]", "tfID": ""}
2024-08-28T04:35:19Z    DEBUG   provider-azure  Reconcile request has been requeued.    {"gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "name": "my-test", "rateLimiterName": "asyncCallback", "when": "5ms"}

Crossplane beta trace for UserAssignedIdentity:

NAME                                               RESOURCE   SYNCED   READY   STATUS
UserAssignedIdentity/user-assigned-identity-test              False    False   ReconcileError: create failed: async create failed: failed to create the resource: [{0 A resource with the ID "/subscriptions/b54258ce-5c95-43bf-8316-1c07ce0b061d/resourceGroups/app-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/user-assigned-identity-test" already exists - to be managed via Terraform this resource needs to be im
ported into the State. Please see the resource documentation for "azurerm_user_assigned_identity" for more information. A resource with the ID "/subscriptions/b54258ce-5c95-43bf-8316-1c07ce0b061d/resourceGroups/app-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/user-assigned-identity-test" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource
 documentation for "azurerm_user_assigned_identity" for more information. []}]

Logs from provider-azure-managedidentity after force deleting pod:

2024-08-28T04:30:35Z    DEBUG   provider-azure  Starting    {"sync-interval": "1h0m0s", "poll-interval": "10m0s", "poll-jitter": "30s", "max-reconcile-rate": 100}
2024-08-28T04:30:37Z    INFO    provider-azure  Beta feature enabled    {"flag": "EnableBetaManagementPolicies"}
2024-08-28T04:30:37Z    DEBUG   provider-azure  Calling the inner handler for Create event. {"gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "name": "user-assigned-identity-test", "queueLength": 0}
2024-08-28T04:30:37Z    DEBUG   provider-azure  Reconciling {"controller": "managed/managedidentity.azure.upbound.io/v1beta1, kind=userassignedidentity", "request": {"name":"user-assigned-identity-test"}}
2024-08-28T04:30:37Z    DEBUG   provider-azure  Connecting to the service provider  {"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:42Z    DEBUG   provider-azure  Instance state not found in cache, reconstructing...    {"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:42Z    DEBUG   provider-azure  Observing the external resource {"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:42Z    DEBUG   provider-azure  Diff detected   {"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"client_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"location\":*terraform.ResourceAttrDiff{Old:\"\", New:\"eastus2\", NewComputed:false, NewRemoved:false, NewExtra:\"eastus2\", RequiresNew:true, Sensitive:false, Type:0x0}, \"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"user-assigned-identity-test\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"principal_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"resource_group_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"app-1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"tenant_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-08-28T04:30:42Z    DEBUG   provider-azure  Async create starting...    {"trackerUID": "292dcb81-22bc-4d20-b2de-142ab1083004", "resourceName": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "tfID": ""}
2024-08-28T04:30:42Z    DEBUG   provider-azure  Creating the external resource  {"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:42Z    DEBUG   provider-azure  Calling the inner handler for Update event. {"gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "name": "user-assigned-identity-test", "queueLength": 0}
2024-08-28T04:30:42Z    DEBUG   provider-azure  Successfully requested creation of external resource    {"controller": "managed/managedidentity.azure.upbound.io/v1beta1, kind=userassignedidentity", "request": {"name":"user-assigned-identity-test"}, "uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "version": "571321", "external-name": "", "external-name": ""}
2024-08-28T04:30:42Z    DEBUG   provider-azure  Reconciling {"controller": "managed/managedidentity.azure.upbound.io/v1beta1, kind=userassignedidentity", "request": {"name":"user-assigned-identity-test"}}
2024-08-28T04:30:42Z    DEBUG   provider-azure  Connecting to the service provider  {"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:43Z    DEBUG   provider-azure  Instance state not found in cache, reconstructing...    {"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:43Z    DEBUG   provider-azure  ongoing async operation {"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "opType": "create"}
2024-08-28T04:30:43Z    DEBUG   provider-azure  External resource is up to date {"controller": "managed/managedidentity.azure.upbound.io/v1beta1, kind=userassignedidentity", "request": {"name":"user-assigned-identity-test"}, "uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "version": "571394", "external-name": "", "requeue-after": "2024-08-28T04:40:30Z"}
2024-08-28T04:30:43Z    DEBUG   provider-azure  Reconciling {"controller": "managed/managedidentity.azure.upbound.io/v1beta1, kind=userassignedidentity", "request": {"name":"user-assigned-identity-test"}}
2024-08-28T04:30:43Z    DEBUG   provider-azure  Connecting to the service provider  {"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:43Z    DEBUG   provider-azure  Async create ended. {"trackerUID": "292dcb81-22bc-4d20-b2de-142ab1083004", "resourceName": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "error": "async create failed: failed to create the resource: [{0 A resource with the ID \"/subscriptions/<SUB_ID>/resourceGroups/app-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/user-assigned-identity-test\" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for \"azurerm_user_assigned_identity\" for more information. A resource with the ID \"/subscriptions/<SUB_ID>/resourceGroups/app-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/user-assigned-identity-test\" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for \"azurerm_user_assigned_identity\" for more information. []}]", "tfID": ""}
2024-08-28T04:30:43Z    DEBUG   provider-azure  Reconcile request has been requeued.    {"gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "name": "user-assigned-identity-test", "rateLimiterName": "asyncCallback", "when": "5ms"}


### Crossplane Version

1.14.3

### Provider Version

1.3.0

### Kubernetes Version

1.28.0

### Kubernetes Distribution

AKS, kind

### Additional Info

_No response_
b-deam commented 2 months ago

I forgot to add, if you just use the default management policies this does not occur.

  managementPolicies:
  - "*"
b-deam commented 2 months ago

May also be related; https://github.com/crossplane-contrib/provider-upjet-azure/issues/796