search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.61k stars 4.65k forks source link

dag/walk infinite loop #22146

Open truekonrads opened 1 year ago

truekonrads commented 1 year ago

Is there an existing issue for this?

Community Note

Terraform Version

1.50

AzureRM Provider Version

3.60.0

Affected Resource(s)/Data Source(s)

azurerm_storage_account, azurerm_storage_accoun

Terraform Configuration Files

resource "azurerm_storage_account" "mainstore" {
  name                     = "${lower(replace(var.global_prefix,"/[^a-z0-9]/",""))}st"
  resource_group_name      = azurerm_resource_group.rg.name
  location                 = azurerm_resource_group.rg.location
  account_tier             = "Standard"
  account_replication_type = "GRS"

  blob_properties {
          change_feed_enabled           = false
        #   change_feed_retention_in_days = 1
          last_access_time_enabled      = false
          versioning_enabled            = false
        }
  timeouts {
    read = "10m"
  }
}

resource "azurerm_role_assignment" "storage_ra1" {
  scope                = azurerm_storage_account.mainstore.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_kubernetes_cluster.aks.identity[0].principal_id
  depends_on = [ 
        azurerm_kubernetes_cluster.aks,
        azurerm_storage_account.mainstore
   ]
}

Debug Output/Panic Output

2023-06-13T15:48:56.639+0800 [TRACE] dag/walk: vertex "root" is waiting for "azurerm_storage_account.mainstore"
2023-06-13T15:48:58.144+0800 [TRACE] dag/walk: vertex "azurerm_role_assignment.storage_ra1 (expand)" is waiting for "azurerm_storage_account.mainstore (expand)"
2023-06-13T15:48:59.797+0800 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2023-06-13T15:49:00.884+0800 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "azurerm_role_assignment.storage_ra1 (expand)"
2023-06-13T15:49:01.655+0800 [TRACE] dag/walk: vertex "root" is waiting for "azurerm_storage_account.mainstore"
2023-06-13T15:49:03.160+0800 [TRACE] dag/walk: vertex "azurerm_role_assignment.storage_ra1 (expand)" is waiting for "azurerm_storage_account.mainstore (expand)"
2023-06-13T15:49:04.812+0800 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2023-06-13T15:49:05.888+0800 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "azurerm_role_assignment.storage_ra1 (expand)"
2023-06-13T15:49:06.669+0800 [TRACE] dag/walk: vertex "root" is waiting for "azurerm_storage_account.mainstore"
2023-06-13T15:49:08.173+0800 [TRACE] dag/walk: vertex "azurerm_role_assignment.storage_ra1 (expand)" is waiting for "azurerm_storage_account.mainstore (expand)"
2023-06-13T15:49:09.825+0800 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2023-06-13T15:49:10.896+0800 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "azurerm_role_assignment.storage_ra1 (expand)"
2023-06-13T15:49:11.683+0800 [TRACE] dag/walk: vertex "root" is waiting for "azurerm_storage_account.mainstore"
2023-06-13T15:49:13.187+0800 [TRACE] dag/walk: vertex "azurerm_role_assignment.storage_ra1 (expand)" is waiting for "azurerm_storage_account.mainstore (expand)"
2023-06-13T15:49:14.839+0800 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2023-06-13T15:49:15.901+0800 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "azurerm_role_assignment.storage_ra1 (expand)"
2023-06-13T15:49:16.694+0800 [TRACE] dag/walk: vertex "root" is waiting for "azurerm_storage_account.mainstore"
2023-06-13T15:49:18.200+0800 [TRACE] dag/walk: vertex "azurerm_role_assignment.storage_ra1 (expand)" is waiting for "azurerm_storage_account.mainstore (expand)"
2023-06-13T15:49:19.855+0800 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2023-06-13T15:49:20.916+0800 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "azurerm_role_assignment.storage_ra1 (expand)"
2023-06-13T15:49:21.713+0800 [TRACE] dag/walk: vertex "root" is waiting for "azurerm_storage_account.mainstore"
2023-06-13T15:49:23.214+0800 [TRACE] dag/walk: vertex "azurerm_role_assignment.storage_ra1 (expand)" is waiting for "azurerm_storage_account.mainstore (expand)"
Stopping operation...
2023-06-13T15:49:24.495+0800 [DEBUG] states/remote: state read serial is: 166; serial is: 166
(I hit Ctrl-C here)
Interrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...

Expected Behaviour

Plan is generated

Actual Behaviour

An infinite loop of two resources waiting on each other - plan isn't generated unless ctrl-c is hit.

Steps to Reproduce

terraform plan

Important Factoids

No response

References

No response

tombuildsstuff commented 1 year ago

hi @truekonrads

Thanks for opening this issue.

Taking a look at the subset of the log provided above, that ordering isn't quite what's happening here - the root vertex is waiting for the Storage Account which is required in order to provision the Role Assignment, which is expected since there's a dependency from the Role Assignment on the Storage Account. It's worth calling out that provisioning a Storage Account can take up to 30 minutes in some cases - and the associated log is from a 30 second window, so based on the information above this looks valid (insofar as the Storage Account is still being provisioned, and the Role Assignment is waiting on this).

As such in order for us to take a look into this issue we'd need the full log output from this Terraform apply, are you able to provide this so that we can take a look?

Thanks!

truekonrads commented 1 year ago

I can try and re-run it and sanitize it. However these are both existing resources with no changes.

webgtx commented 1 year ago

I faced similar issue after I've changed azure subscription

every time I run terraform plan with Microsoft Sponsorhip Subscription I getting the same ininite dag/walk in tracelog, but when I switch to default Azure Subscription everything works fine

2023-10-30T09:00:23.490Z [TRACE] dag/walk: vertex "azurerm_kubernetes_cluster_node_pool.frontend (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:23.490Z [TRACE] dag/walk: vertex "root" is waiting for "output.kube_config (expand)"
2023-10-30T09:00:23.490Z [TRACE] dag/walk: vertex "azurerm_kubernetes_cluster.stage (expand)" is waiting for "azurerm_resource_group.careerist (expand)"
2023-10-30T09:00:23.490Z [TRACE] dag/walk: vertex "output.client_certificate (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:23.491Z [TRACE] dag/walk: vertex "azurerm_resource_group.careerist (expand)" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"]"
2023-10-30T09:00:23.491Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "azurerm_kubernetes_cluster_node_pool.frontend (expand)"
2023-10-30T09:00:23.491Z [TRACE] dag/walk: vertex "output.kube_config (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:23.491Z [TRACE] dag/walk: vertex "azurerm_kubernetes_cluster_node_pool.skillsapi (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:28.492Z [TRACE] dag/walk: vertex "azurerm_kubernetes_cluster_node_pool.skillsapi (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:28.492Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "azurerm_kubernetes_cluster_node_pool.frontend (expand)"
2023-10-30T09:00:28.492Z [TRACE] dag/walk: vertex "azurerm_kubernetes_cluster_node_pool.frontend (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:28.492Z [TRACE] dag/walk: vertex "output.client_certificate (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:28.492Z [TRACE] dag/walk: vertex "root" is waiting for "output.kube_config (expand)"
2023-10-30T09:00:28.492Z [TRACE] dag/walk: vertex "azurerm_kubernetes_cluster.stage (expand)" is waiting for "azurerm_resource_group.careerist (expand)"
2023-10-30T09:00:28.492Z [TRACE] dag/walk: vertex "output.kube_config (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:28.492Z [TRACE] dag/walk: vertex "azurerm_resource_group.careerist (expand)" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"]"
2023-10-30T09:00:29.774Z [DEBUG] provider.terraform-provider-azurerm_v3.77.0_x5: AzureRM Request: 
GET /subscriptions/425****************d04d6ed13d/providers/Microsoft.Web?api-version=2022-09-01 HTTP/1.1
Host: management.azure.com
User-Agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 providers/2022-09-01) HashiCorp Terraform/1.5.7 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.77.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Accept: application/json; charset=utf-8; IEEE754Compatible=false
Content-Type: application/json; charset=utf-8
Odata-Maxversion: 4.0
Odata-Version: 4.0
X-Ms-Correlation-Request-Id: **********-f882-feb1-e*****1cfe7
Accept-Encoding: gzip: timestamp=2023-10-30T09:00:29.774Z
2023-10-30T09:00:29.774Z [DEBUG] provider.terraform-provider-azurerm_v3.77.0_x5: GET https://management.azure.com/subscriptions/**************98d04***3d/providers/Microsoft.Web?api-version=2022-09-01: timestamp=2023-10-30T09:00:29.774Z
2023-10-30T09:00:33.493Z [TRACE] dag/walk: vertex "azurerm_kubernetes_cluster_node_pool.skillsapi (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:33.493Z [TRACE] dag/walk: vertex "azurerm_resource_group.careerist (expand)" is waiting for "provider[\"registry.terraform.io/hashicorp/azurerm\"]"
2023-10-30T09:00:33.493Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "azurerm_kubernetes_cluster_node_pool.frontend (expand)"
2023-10-30T09:00:33.493Z [TRACE] dag/walk: vertex "azurerm_kubernetes_cluster_node_pool.frontend (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:33.493Z [TRACE] dag/walk: vertex "output.client_certificate (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
2023-10-30T09:00:33.493Z [TRACE] dag/walk: vertex "root" is waiting for "output.kube_config (expand)"
2023-10-30T09:00:33.493Z [TRACE] dag/walk: vertex "azurerm_kubernetes_cluster.stage (expand)" is waiting for "azurerm_resource_group.careerist (expand)"
2023-10-30T09:00:33.493Z [TRACE] dag/walk: vertex "output.kube_config (expand)" is waiting for "azurerm_kubernetes_cluster.stage (expand)"
MattCosturos commented 1 year ago

I had the same issue, an endless loop of dag/walk: vertex.

I was running the plan in an Azure pipeline. I tested locally. I hit Ctrl-C to kill the process locally. Finally, I found the true source of the error. In my case, I was not using an act that has permissions to register resource providers.

Error: Error ensuring Resource Providers are registered.
│ 
│ Terraform automatically attempts to register the Resource Providers it supports to
│ ensure it's able to provision resources.
│ 
│ If you don't have permission to register Resource Providers you may wish to use the
│ "skip_provider_registration" flag in the Provider block to disable this functionality.
│ 
│ Please note that if you opt out of Resource Provider Registration and Terraform tries
│ to provision a resource from a Resource Provider which is unregistered, then the errors
│ may appear misleading - for example: