[BUG] Updating from 1.12.1 to 1.13.0 causes forced replacement on storage blob container and keyvault key due to "location=<LOC>->null"

[Brakistad]

Summary of Issues with azapi Provider

Description: After updating the azapi provider from version 1.12.1 to 1.13.0 and Terraform itself, we encountered issues with forced replacements of resources, even though there are no significant changes to the configuration.

Error Messages:

1. Forced Resource Replacements:

   • Terraform plans indicate forced replacements of resources due to the location attribute being set to null. This behavior affects both blob containers and Key Vault keys.

     -/+ resource "azapi_resource" "key_vault_key" {
        ~ id                        = "/subscriptions/example/resourceGroups/example-rg/providers/Microsoft.KeyVault/vaults/examplekv/keys/ExampleKey" -> (known after apply)
        - location                  = "norwayeast" -> null # forces replacement
          name                      = "ExampleKey"
        ~ output                    = jsonencode({}) -> (known after apply)
        - tags                      = {} -> null
          # (other unchanged attributes)
      }
     
     -/+ resource "azapi_resource" "blob_container" {
        ~ id                        = "/subscriptions/example/resourceGroups/example-rg/providers/Microsoft.Storage/storageAccounts/exampleaccount/blobServices/default/containers/examplecontainer" -> (known after apply)
        - location                  = "norwayeast" -> null # forces replacement
          name                      = "examplecontainer"
        ~ output                    = jsonencode({}) -> (known after apply)
        - tags                      = {} -> null
          # (other unchanged attributes)
      }

Impact:

• The forced replacements of resources cause unnecessary destruction and recreation, which can lead to service disruption.

Steps to Reproduce:

1. Create a keyvault key and a blob container while on version 1.12.1
2. Update the azapi provider from version 1.12.1 to 1.13.0.
3. Run terraform plan with a configuration that includes azapi resources for blob containers and Key Vault keys.

[ms-henglu]

Hi @Brakistad ,

Thank you for taking time to report this issue!

In previous version, the location might not be correctly reflected in terraform state. Would you please add the location field to the resource block to match with remote state?

And I'd recommend to use v1.13.1 instead of v1.13.0 because v1.13.1 contains multiple bug fixes.

[Brakistad]

Hi @ms-henglu,

Thank you for your previous response and guidance.

Current Status:

After updating the azapi provider to version 1.13.1, we have modified our configuration to include schema_validation_enabled = false for the affected resources. Here is an example of our updated configuration for the Key Vault Key resource:

# Key Vault Key
resource "azapi_resource" "key_vault_key" {
  for_each                  = { for k, v in var.principal : k => v if v != "" }
  type                      = "Microsoft.KeyVault/vaults/keys@2023-07-01"
  name                      = "EncryptionKey"
  schema_validation_enabled = false
  parent_id                 = <PARENT ID>
  location                  = var.location
  body = jsonencode({
    properties = {
       ... valid properties
    }
  })
}

Plan Output:

Upon running terraform plan, the output indicates that all changes are updates in place, and no resources will be recreated, which seems promising.

Concerns:

While the plan step appears to validate the configuration correctly, we are concerned about the apply step. Specifically, our concern is that the apply step might encounter issues if the Azure API call fails due to schema validation being disabled. This would result in failed deployments and would require us to revert the changes, bringing us back to the initial issue with forced replacements.

Could you please confirm if the current fix using schema_validation_enabled = false is the intended solution, and if there are any additional steps we should take to ensure successful application of the configuration?

Thank you for your support and guidance.

EDIT:

Additional information: We had to use 'schema_validation_enabled = false' to not break in the plan step, because the provider did not recognize the parameter "location" in the azapi_resource block.

[ms-henglu]

Thanks for the quick response!

Yes, you could the schema_validation_enabled = false. The schema validation is applied on the client side, so disabling it will not affect the API calls.

I'll check the if the swagger has incorrect definition and get back to you.

[Brakistad]

Hi @ms-henglu,

Thank you for your previous support and guidance.

Current Status:

We have successfully updated the azapi provider to version 1.13.1 and applied the fix using schema_validation_enabled = false, which has resolved the initial issue of forced replacements. However, we are now encountering a new problem related to tag changes. Specifically, Terraform's plan step is consistently showing in-place updates due to tag changes, as seen in the following examples:

# Example of tag changes causing in-place updates
# module.submoduleX["id"].azapi_resource.blob_container["id"] will be updated in-place
~ resource "azapi_resource" "blobcontainer" {
    id = "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/example-rg/providers/Microsoft.Storage/storageAccounts/example-sa/blobServices/default/containers/blob-example"
    name = "blobcontainer"
    ~ output = jsonencode({}) -> (known after apply)
    + tags = {
        + "BusinessUnit" = "BUSINESS_UNIT"
        + "CostCenter" = "COST_CENTER"
        + "Criticality" = "CRITICALITY_LEVEL"
        + "DataClassification" = "DATA_CLASSIFICATION"
        + "EndDate" = "END_DATE"
        + "Env" = "ENVIRONMENT"
        + "ManagedBy" = "MANAGED_BY"
        + "OpsCommitment" = "OPS_COMMITMENT"
        + "OpsTeam" = "OPS_TEAM"
        + "Owner" = "OWNER_EMAIL"
        + "Requester" = "REQUESTER_EMAIL"
        + "StartDate" = "START_DATE"
        + "WorkloadName" = "WORKLOAD_NAME"
    }
    # (8 unchanged attributes hidden)
}

# Similar updates are shown for other resources

Issue:

• Repeated Tag Changes: The plan shows updates for adding tags, even when the tags are already set in the configuration. This results in repeated "in-place" updates during every plan and apply cycle.
• Behavior with Explicit Tag Setting: If we explicitly set the tags in the Terraform configuration, the plan shows additions. If we remove the explicit tag settings, the plan indicates tag removals. This creates a loop where either tags are added or removed in each plan step.

Concerns:

We are concerned that this behavior will lead to unnecessary updates and noise during deployment. We would like to confirm if this behavior is expected with the current provider version or if there are any additional steps we should take to stabilize the tag management.

[ms-henglu]

Thanks for the details. But these tags are not introduced by azapi provider, please check whether there're some policy or automation flow that adds tags to your resource.

You could check the activity logs to see who edited the resource.

[rvdouderaa]

We ran into this issue using Microsoft.web/sites/siteextensions, created with version 1.13.1 and doing a rerun of the configuration. So the orginal deployment was done using 1.13.1 as well.

The rerun want's to replace location from West Europe to null. The location was not set using the deployment (since this is not a property of the resource https://learn.microsoft.com/en-us/azure/templates/microsoft.web/sites/siteextensions?pivots=deployment-language-terraform)

The change of the location forces a redeployment of the resources, which isn't possible on a running web app due to the nature of the extension.

We tried running this using provider version 1.12.1 which 'works like a charm`.

[rvdouderaa]

@ms-henglu any updates on this?

[ms-henglu]

Hi @rvdouderaa , sorry for late response.

I think it's a similar case, please check: https://github.com/Azure/terraform-provider-azapi/issues/514#issuecomment-2126606875

[rvdouderaa]

Hi @ms-henglu

afaik location is nog accepted as a property for the siteextensions. So, Terraform does want to change it, but you cannot set it.

[ms-henglu]

I see. Another workaround is adding the lifecycle to suppress the diff.

[nmehlei]

I managed to fix it for me with adding schema_validation_enabled = false as well as the lifecycle/ignore_changes setting. Though I needed to manually delete the extension and then re-create it again with schema_validation_enabled = false, as trying to modify the deployed resource to change schema_validation_enabled led to errors.

resource "azapi_resource" "newrelic_site_extension" {
    type = "Microsoft.Web/sites/siteextensions@2022-03-01"
    name = "NewRelic.Azure.WebSites.Extension"
    parent_id = azurerm_windows_web_app.appservice.id
    schema_validation_enabled = false
    depends_on = [ azurerm_windows_web_app.appservice ]

    lifecycle {
        ignore_changes = [
            location,
            id
        ]
    }
}

Disclaimer: I did not do extensive testing on it. There is a chance it may break on update attempts if there are other diff's that would enforce re-creation.