cannot create AKS cluster in TFC

ausfestivus commented 5 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

😀 abest@BARMIX2:~/github/vault-aks-terraform (develop) $ terraform -v
Terraform v0.12.12
+ provider.azurerm v1.35.0

Affected Resource(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

resource "azurerm_kubernetes_cluster" "vault-aks-terraform" {
    name = var.prefix
    location = azurerm_resource_group.rg.location
    resource_group_name = azurerm_resource_group.rg.name
    dns_prefix = "${var.prefix}-rbac"

    agent_pool_profile {
        name            = "default"
        count           = 1
        vm_size         = "Standard_D1_v2"
        os_type         = "Linux"
        os_disk_size_gb = 30
    }

    service_principal {
        client_id     = var.kubernetes_client_id
        client_secret = var.kubernetes_client_secret
    }

    role_based_access_control {
        enabled = true

        azure_active_directory {
            # NOTE: in a Production environment these should be different values
            # but for the purposes of this example, this should be sufficient
            client_app_id = var.kubernetes_client_id

            server_app_id     = var.kubernetes_client_id
            server_app_secret = var.kubernetes_client_secret
        }
    }
    tags     = var.tags
}

Debug Output

https://gist.github.com/ausfestivus/95923bfd17a7af0c7f54f8faa4b205a6

Panic Output

nil

Expected Behavior

AKS Cluster created via TFC.

Actual Behavior

The following error is displayed during the apply.

Terraform v0.12.12
Initializing plugins and modules...
2019/10/26 06:08:03 [DEBUG] Using modified User-Agent: Terraform/0.12.12 TFC/623b46863f
azurerm_kubernetes_cluster.vault-aks-terraform: Creating...

Error: Error creating Managed Kubernetes Cluster "vault-aks-terraform" (Resource Group "vault-aks-terraform-develop-rg"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="BadRequest" Message="The access token requested for audience https://graph.microsoft.com by application 554a5294-f9bd-495e-9dbe-5de320fd9858 in tenant a5aa424e-5d6f-47c9-bf70-a1310f4be302 is missing the required claim role Directory.Read.All." Target="aadProfile.serverAppID"

  on aks.tf line 8, in resource "azurerm_kubernetes_cluster" "vault-aks-terraform":
   8: resource "azurerm_kubernetes_cluster" "vault-aks-terraform" {

Steps to Reproduce

terraform apply

Important Factoids

If I login to the Azure CLI using the TFC SP and run the az aks create ... command the AKS cluster is created.
There are two Azure Subscriptions here. Both attached to the same AAD. One is considered pre-prod, the other is my MSDN Subscription.
The SP used by TFC is able to create other TF builds in either Subscription.
The SP that azurerm_kubernetes_cluster is using for RBAC is in the same AAD as the TFC SP.

References

AKS doco for SPs AzureRM Provider Example used for azurerm_kubernetes_cluster

Edits

Add Factoid 1. It's an important one duh? 🙄

0000

tombuildsstuff commented 5 years ago

hi @ausfestivus

Taking a look at the error being returned from the Azure API here:

The access token requested for audience https://graph.microsoft.com by application 554a5294-f9bd-495e-9dbe-5de320fd9858 in tenant a5aa424e-5d6f-47c9-bf70-a1310f4be302 is missing the required claim role Directory.Read.All.

It appears the Service Principal being used doesn't have permission to read items in Azure Active Directory, which is happening when the AKS API is validating the Service Principal prior to using it. If this is a new Service Principal there's an additional step required after creating the Service Principal outlined here: https://www.terraform.io/docs/providers/azuread/auth/service_principal_configuration.html - would you be able to take a look and see if that works for you?

Since this is a question regarding Terraform Configuration rather than a bug in the Azure Provider I'm going to close this issue for the moment, but please let us know if that doesn't work for you and we'll take another look.

Thanks!

ausfestivus commented 5 years ago

Hey Tom,

Appreciate the response. I did what you advised and have not had any change in behaviour.

If I run the build from TFC I get the error. (TFC uses an SP to talk to Azure [that SP works for other TFC builds]. The Kube SP is present.) If I run the build using the Azure CLI it works. I am using the example described at https://github.com/terraform-providers/terraform-provider-azurerm/tree/master/examples/kubernetes/role-based-access-control-azuread.

I cannot see what im missing here and im unsure how to really get down lower to divine what's happening.

ausfestivus commented 5 years ago

Sorry @tombuildsstuff I didnt tag you in the above response so you may not have seen it.

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

hashicorp / terraform-provider-azurerm