azurerm_storage_container entries are disappearing from the state file during Terraform plan/apply

aatuch commented 4 weeks ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.5.4

AzureRM Provider Version

3.107.0

Affected Resource(s)/Data Source(s)

azurerm_storage_container

Terraform Configuration Files

resource "azurerm_storage_container" "cicd-container" {
  provider              = azurerm.mockcustomprovidername
  name                  = "cicd"
  storage_account_name  = "mockstorageaccountname"
  container_access_type = "private"
}

resource "azurerm_storage_container" "mtt-container" {
  provider              = azurerm.mockcustomprovidername
  name                  = "mtt"
  storage_account_name  = "mockstorageaccountname"
  container_access_type = "private"
}

Debug Output/Panic Output

2024-06-12T10:21:45.6003436Z azurerm_storage_container.mtt-container: Refreshing state... [id=https://mockstorageaccountname.blob.core.windows.net/mtt]
2024-06-12T10:21:45.6003819Z azurerm_storage_container.cicd-container: Refreshing state... [id=https://mockstorageaccountname.blob.core.windows.net/cicd]
2024-06-12T10:21:46.8668250Z 
2024-06-12T10:21:46.8668695Z Note: Objects have changed outside of Terraform
2024-06-12T10:21:46.8668765Z 
2024-06-12T10:21:46.8668902Z Terraform detected the following changes made outside of Terraform since the
2024-06-12T10:21:46.8669066Z last "terraform apply" which may have affected this plan:
2024-06-12T10:21:46.8669134Z 
2024-06-12T10:21:46.8669527Z   # azurerm_storage_container.cicd-container has been deleted
2024-06-12T10:21:46.8669746Z   - resource "azurerm_storage_container" "cicd-container" {
2024-06-12T10:21:46.8669900Z         id                                = "https://mockstorageaccountname.blob.core.windows.net/cicd"
2024-06-12T10:21:46.8670032Z         name                              = "cicd"
2024-06-12T10:21:46.8670711Z       - resource_manager_id               = "/subscriptions/11111111-2222-3333-4444-555555555555/resourceGroups/mockname-rg/providers/Microsoft.Storage/storageAccounts/mockstorageaccountname/blobServices/default/containers/cicd" -> null
2024-06-12T10:21:46.8670977Z         # (7 unchanged attributes hidden)
2024-06-12T10:21:46.8671149Z     }
2024-06-12T10:21:46.8671216Z 
2024-06-12T10:21:46.8671401Z   # azurerm_storage_container.mtt-container has been deleted
2024-06-12T10:21:46.8671611Z   - resource "azurerm_storage_container" "mtt-container" {
2024-06-12T10:21:46.8671761Z         id                                = "https://mockstorageaccountname.blob.core.windows.net/mtt"
2024-06-12T10:21:46.8671957Z         name                              = "mtt"
2024-06-12T10:21:46.8672570Z       - resource_manager_id               = "/subscriptions/11111111-2222-3333-4444-555555555555/resourceGroups/mockname-rg/providers/Microsoft.Storage/storageAccounts/mockstorageaccountname/blobServices/default/containers/mtt" -> null
2024-06-12T10:21:46.8672901Z         # (7 unchanged attributes hidden)
2024-06-12T10:21:46.8673027Z     }
2024-06-12T10:21:46.8673063Z 
2024-06-12T10:21:46.8673156Z 
2024-06-12T10:21:46.8673298Z Unless you have made equivalent changes to your configuration, or ignored the
2024-06-12T10:21:46.8673521Z relevant attributes using ignore_changes, the following plan may include
2024-06-12T10:21:46.8673724Z actions to undo or respond to these changes.
2024-06-12T10:21:46.8673830Z 
2024-06-12T10:21:46.8674379Z ─────────────────────────────────────────────────────────────────────────────
2024-06-12T10:21:46.8674675Z 
2024-06-12T10:21:46.8674907Z Terraform used the selected providers to generate the following execution
2024-06-12T10:21:46.8675125Z plan. Resource actions are indicated with the following symbols:
2024-06-12T10:21:46.8675304Z   + create
2024-06-12T10:21:46.8675356Z 
2024-06-12T10:21:46.8675481Z Terraform will perform the following actions:
2024-06-12T10:21:46.8675597Z 
2024-06-12T10:21:46.8675804Z   # azurerm_storage_container.cicd-container will be created
2024-06-12T10:21:46.8676014Z   + resource "azurerm_storage_container" "cicd-container" {
2024-06-12T10:21:46.8676165Z       + container_access_type             = "private"
2024-06-12T10:21:46.8676303Z       + default_encryption_scope          = (known after apply)
2024-06-12T10:21:46.8676437Z       + encryption_scope_override_enabled = true
2024-06-12T10:21:46.8676577Z       + has_immutability_policy           = (known after apply)
2024-06-12T10:21:46.8676772Z       + has_legal_hold                    = (known after apply)
2024-06-12T10:21:46.8677157Z       + id                                = (known after apply)
2024-06-12T10:21:46.8677337Z       + metadata                          = (known after apply)
2024-06-12T10:21:46.8677466Z       + name                              = "cicd"
2024-06-12T10:21:46.8677632Z       + resource_manager_id               = (known after apply)
2024-06-12T10:21:46.8677818Z       + storage_account_name              = "mockstorageaccountname"
2024-06-12T10:21:46.8677942Z     }
2024-06-12T10:21:46.8677990Z 
2024-06-12T10:21:46.8678197Z   # azurerm_storage_container.mtt-container will be created
2024-06-12T10:21:46.8678591Z   + resource "azurerm_storage_container" "mtt-container" {
2024-06-12T10:21:46.8678788Z       + container_access_type             = "private"
2024-06-12T10:21:46.8678978Z       + default_encryption_scope          = (known after apply)
2024-06-12T10:21:46.8679159Z       + encryption_scope_override_enabled = true
2024-06-12T10:21:46.8679349Z       + has_immutability_policy           = (known after apply)
2024-06-12T10:21:46.8679540Z       + has_legal_hold                    = (known after apply)
2024-06-12T10:21:46.8679705Z       + id                                = (known after apply)
2024-06-12T10:21:46.8679881Z       + metadata                          = (known after apply)
2024-06-12T10:21:46.8680010Z       + name                              = "mtt"
2024-06-12T10:21:46.8680125Z       + resource_manager_id               = (known after apply)
2024-06-12T10:21:46.8680309Z       + storage_account_name              = "mockstorageaccountname"
2024-06-12T10:21:46.8680474Z     }
2024-06-12T10:21:46.8680524Z 
2024-06-12T10:21:46.8680668Z Plan: 2 to add, 0 to change, 0 to destroy.

Expected Behaviour

Our Terraform configuration has been working without errors for a year, and we have not made any changes to the containers. Terraform should run and output "No changes. Your infrastructure matches the configuration."

Actual Behaviour

In the last week the issue has occurred 4 times on different environments at random times among around 40 deployments to different environments. During deploy container entries in the state file are suddenly deleted. Terraform then tries to create resources that are not in the state and fails with an error when it encounters them in Azure. It was mitigated by importing the container resources. On one environment it reproduced even after the import. Any external or human interference is excluded. The deployment is exclusively run from Azure DevOps pipelines.

Steps to Reproduce

terraform plan or terraform apply

The issue is transient. I cannot reproduce it by just redeploying. It happens randomly. No patterns have been observed.

Important Factoids

No response

References

No response

magodo commented 3 weeks ago

@aatuch Could you please provide the debug log when the issue happens so that we can better understand what caused TF regard the containers as absent. Back in the provider code, there are two cases this will happen:

Failed to locate the storage account:

https://github.com/hashicorp/terraform-provider-azurerm/blob/fa66cdbffb9817b930fd11070f216d016f58e326/internal/services/storage/storage_container_resource.go#L255-L263
Failed to get the container

https://github.com/hashicorp/terraform-provider-azurerm/blob/fa66cdbffb9817b930fd11070f216d016f58e326/internal/services/storage/storage_container_resource.go#L270-L278

aatuch commented 3 weeks ago

@magodo, unfortunately, I cannot provide the debug log since I cannot reproduce the issue.

tombuildsstuff commented 3 weeks ago

@aatuch out of interest, are you using the Azure CLI to authenticate?

hashicorp / terraform-provider-azurerm