azurerm_kubernetes_cluster: roleAssignment permission needed when http_application_routing addon is explicitly disabled

invidian commented 5 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

$ terraform version
Terraform v0.11.14
+ provider.azurerm v1.30.1

Affected Resource(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

  addon_profile {
    http_application_routing {
      enabled = false
    }
  }

Debug Output

Panic Output

Expected Behavior

Cluster creation shouldn't require any additional privileges and cluster creation should succeed.

Note: with mentioned block removed completely, creation of the cluster obviously works and the addon is disabled by default.

Actual Behavior

Terraform fails to create AKS cluster with following error message:

1 error occurred:
    * azurerm_kubernetes_cluster.test: 1 error occurred:
    * azurerm_kubernetes_cluster.test: Error waiting for completion of Managed Kubernetes Cluster "acctestaks1" (Resource Group "'<removed>"): Code="CreateRoleAssignmentError" Message="RoleAssignmentReconciler retry timed out: autorest/azure: Service returned an error. Status=403 Code=\"AuthorizationFailed\" Message=\"The client '<removed>' with object id '<removed>' does not have authorization to perform action 'Microsoft.Authorization/roleAssignments/write' over scope '/subscriptions/<removed>/resourceGroups/<removed>/providers/Microsoft.Authorization/roleAssignments/<removed>' or the scope is invalid. If access was recently granted, please refresh your credentials.\""

Steps to Reproduce

terraform apply

Important Factoids

References

0000

invidian commented 5 years ago

Okay, now it gets really confusing. This works as expected:

  addon_profile {
    http_application_routing {
      enabled = true
    }
  }

And when I change it back to false, it recreates the cluster and then it works.

patst commented 5 years ago

Seems to be the same as https://github.com/terraform-providers/terraform-provider-azurerm/issues/3912

I'm getting the same error using a config which worked for weeks. After waiting a minute and trying again it works. Seems like some internal permissions are not propagated fast enough...

Using az ad sp show --id <objectId-from-error> a app named 'AzureContainerService' is returned belonging to a tenant different then the one I'm using.

neil-yechenwei commented 4 years ago

Thanks for opening this issue. After tested, seems aks cluster can be created successfully. Seems I cannot repro the issue with below config. Maybe I missed something. If you need, you can try below config and check whether the problem still exists.

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name     = "acctestRG-aks-neil"
  location = "westus2"
}

resource "azurerm_kubernetes_cluster" "test" {
  name                = "acctestaksneil"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  dns_prefix          = "acctestaksneil"

  linux_profile {
    admin_username = "acctestuserneil"

    ssh_key {
      key_data = "<insert ssh key>"
    }
  }

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_DS2_v2"
  }

  addon_profile {
    http_application_routing {
      enabled = false
    }
  }

  identity {
    type = "SystemAssigned"
  }
}

Result:

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

invidian commented 4 years ago

Hey @neil-yechenwei, I'm not sure if I find time for testing again this issue. Maybe there were some changes after this has been reported. Maybe what's important is during reproducing, that this needs to be tested with relatively not-privileged service account, e.g. with Contributor role on subscription level or resource group level. I guess if you have some privileges on Azure AD level, then this might just succeed.

neil-yechenwei commented 4 years ago

I think it's the similar issue with #3912. So it should be already fixed. If you still encounter same issue, please expose it. Thanks.

mybayern1974 commented 4 years ago

Hi @invidian , per the discussion above would you mind I close this issue for now? Feel free to reopen once it happens again. Thank you!

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

hashicorp / terraform-provider-azurerm