search

microsoft / terraform-provider-azuredevops

Terraform Azure DevOps provider
https://www.terraform.io/docs/providers/azuredevops/
MIT License
387 stars 278 forks source link

azuredevops_serviceendpoint_azurecr failing to destroy #1131

Open bwesolowskicf opened 3 months ago

bwesolowskicf commented 3 months ago

Community Note

Terraform (and Azure DevOps Provider) Version

Terraform v1.9.0 microsoft/azuredevops v1.2.0

Affected Resource(s)

Terraform Configuration Files

resource "azurerm_user_assigned_identity" "devops_acr" {
  location            = data.azurerm_resource_group.identity.location
  name                = "identity-shared-devops-acr-${var.app_short_name}"
  resource_group_name = data.azurerm_resource_group.identity.name
}

resource "azuredevops_serviceendpoint_azurecr" "acr" {
  project_id                             = local.azuredevops_project.id
  service_endpoint_name                  = "DOCKER-ACR-${var.app_short_name}"
  resource_group                         = "RG"
  service_endpoint_authentication_scheme = "WorkloadIdentityFederation"
  azurecr_spn_tenantid                   = "TENANT_ID"
  azurecr_name                           = "ACRNAME"
  azurecr_subscription_id                = "SUB_ID"
  azurecr_subscription_name              = "SUB_NAME"
  credentials {
    serviceprincipalid = azurerm_user_assigned_identity.devops_acr.client_id
  }
}

resource "azurerm_federated_identity_credential" "acr_identity_credential" {
  depends_on          = [azuredevops_serviceendpoint_azurecr.acr]
  name                = "acr_identity_credential_${var.app_short_name}"
  resource_group_name = azurerm_user_assigned_identity.devops_acr.resource_group_name
  parent_id           = azurerm_user_assigned_identity.devops_acr.id
  audience            = ["api://AzureADTokenExchange"]
  issuer              = azuredevops_serviceendpoint_azurecr.acr.workload_identity_federation_issuer
  subject             = azuredevops_serviceendpoint_azurecr.acr.workload_identity_federation_subject
}

Debug Output

Panic Output

Expected Behavior

We have yaml files for team-specific configuration and we create resources such as Azure DevOps project per such yaml file. In our QA pipeline we have tests for creating new resources and (since we had some issues with terraform destroy in the past) we achieve that by deploying all resources, running some tests and then we delete the yaml file and run terraform apply. That way all "Temp" resources are deleted and on next run we can recreate them again.

What we expect here is for the service endpoint to be deleted.

Actual Behavior

We get an error.

Error:  Delete service endpoint error Cannot delete manually created Workload Identity Federation Docker Registry service connection while related App Registration <ID OF THE CREATED IDENTITY> exists in Azure under tenant <OUR AZURE TENANT ID>.

Steps to Reproduce

  1. Put the above code as a module and invoke the module in main.tf
  2. terraform apply
  3. Remove the module invocation from main.tf
  4. terraform apply - this will try to remove all resources and fail on the service connection

Important Factoids

References

xuzhang3 commented 2 months ago

@bwesolowskicf This is controlled by service. Service will check the related resources in the AAD when trying to delete the service connection.

bwesolowskicf commented 2 months ago

@xuzhang3 Hi, I'm not quite sure what does 'controlled by service' mean. Is there anything I can do to resolve this issue?

xuzhang3 commented 2 months ago

@bwesolowskicf ADO service will check the related app status in AAD when you trying to delete azuredevops_serviceendpoint_azurecr which means that if the application (azurerm_federated_identity_credential) still exists, the deletion will be rejected. You can add time_sleep to wait a moment after azurerm_federated_identity_credential deleted

karts499 commented 3 weeks ago

unfortunately the issue is still there. I have tried to delete the same resource as above. The only workaround I have found working for me is to delete the secret via the portal or do a terraform destroy -target before the main task. @xuzhang3 time_sleep wont work for destroy. the main problem is that terraform tries to the destroy the azuredevops_serviceendpoint_azurecr first and not the azurerm_federated_identity_credential. As far as I know there is no way to influence the destruction sequence. Not yet...

xuzhang3 commented 3 weeks ago

@karts499 Have you tried depends_on?

Fresa commented 3 days ago

Same problem with azuredevops_serviceendpoint_azurerm. Works when running a second time, which might be due to eventual consistency I guess. Wouldn't it make sense that the provider retries a couple of times if this is expected?

karts499 commented 1 day ago

@karts499 Have you tried depends_on? yes I did, wont change the outcome. You simply can´t change the destroy order. depends_on just works as intented for the creation. For destruction terraform just tries to delete both not simultaneously but in the wrong order...

xuzhang3 commented 10 hours ago

Normally when a new Azure Container Registry or AzureRM service endpoint created, a new APP will be created the the AAD. In my test env I can can create new Azure Container registry or AzureRM service endpoint and destroy it with no errors, although the APP still exists in ADD. In some cases, when we destroy the service connection, the service also tries to destroy the application but I cannot reproduce error.

App in MS Entra: Image