search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.28k stars 21.45k forks source link

It's now possible to setup RBAC, but it's not clear how. #20862

Closed jpetitte closed 5 years ago

jpetitte commented 5 years ago

Setting up AKS with RBAC via Terraform is now possible, but it's not clear to me exactly how. I've also run across a few other folks in various GitHub issues struggling with the same thing.

It seems like the key is to implement https://docs.microsoft.com/en-us/azure/aks/aad-integration using Terraform, but there are some steps that do not obviously translate into AzureRM Terraform config. (for example: 'Select Manifest and edit the groupMembershipClaims value to "All".')

I've got an implementation that seems to be partially correct (I can post if you would like), but I'm ending up with 'serviceaccounts is forbidden: User "system:anonymous" cannot create serviceaccounts in the namespace "kube-system"', and I'm having trouble determining if this is related to my incorrect configuration of K8s with AzureRM or something else. Is there some guidance on correctly setting up RBAC in AKS using Terraform?

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

Karishma-Tiwari-MSFT commented 5 years ago

Thanks for the question. We are currently investigating and will update you shortly.

mimckitt commented 5 years ago

@jpetitte please feel free to post the solution you found :) I will assign to the content author to review and see if we can get more steps added around this.

jpetitte commented 5 years ago

@MicahMcKittrick-MSFT To be clear, I'm having problems with the configuration below (as described above), but hopefully, this is close. I've also added my attempt to get Tiller setup in the cluster as well (maybe it will help someone else).

# Configure the Azure Provider
provider "azurerm" {}

resource "azurerm_resource_group" "rg" {
  name     = "${var.resource_group_name}"
  location = "${var.location}"

  # tags = {
  #   workspace = "myapp"
  # }
}

resource "azurerm_log_analytics_workspace" "la_workspace" {
  name                = "${var.log_analytics_workspace_name}"
  location            = "${var.log_analytics_workspace_location}"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  sku                 = "${var.log_analytics_workspace_sku}"
}

resource "azurerm_log_analytics_solution" "la_solution" {
  solution_name         = "ContainerInsights"
  location              = "${azurerm_log_analytics_workspace.la_workspace.location}"
  resource_group_name   = "${azurerm_resource_group.rg.name}"
  workspace_resource_id = "${azurerm_log_analytics_workspace.la_workspace.id}"
  workspace_name        = "${azurerm_log_analytics_workspace.la_workspace.name}"

  plan {
    publisher = "Microsoft"
    product   = "OMSGallery/ContainerInsights"
  }
}

# acr
resource "azurerm_storage_account" "storage" {
  name                     = "myappacrstorage"
  resource_group_name      = "${azurerm_resource_group.rg.name}"
  location                 = "${azurerm_resource_group.rg.location}"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_container_registry" "acr" {
  name                = "myappacr"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  location            = "${azurerm_resource_group.rg.location}"

  # admin_enabled       = true
  sku                = "Classic"
  storage_account_id = "${azurerm_storage_account.storage.id}"
}

# rbac
resource "azurerm_azuread_application" "sp" {
  name                       = "myapp-sp"
  homepage                   = "https://myapp-sp"
  identifier_uris            = ["https://myapp-sp"]
  reply_urls                 = ["https://myapp-sp"]
  available_to_other_tenants = false
  oauth2_allow_implicit_flow = false
}

resource "azurerm_azuread_application" "server" {
  name                       = "myapp-server"
  homepage                   = "https://myapp-server"
  identifier_uris            = ["https://myapp-server"]
  reply_urls                 = ["https://myapp-server"]
  available_to_other_tenants = false
  oauth2_allow_implicit_flow = false
}

resource "azurerm_azuread_application" "client" {
  name                       = "myapp-client"
  homepage                   = "https://myapp-client"
  identifier_uris            = ["https://myapp-client"]
  reply_urls                 = ["https://myapp-client"]
  available_to_other_tenants = false
  oauth2_allow_implicit_flow = false
}

resource "azurerm_azuread_service_principal" "sp" {
  application_id = "${azurerm_azuread_application.sp.application_id}"
}

resource "azurerm_azuread_service_principal" "server" {
  application_id = "${azurerm_azuread_application.server.application_id}"
}

resource "azurerm_azuread_service_principal" "client" {
  application_id = "${azurerm_azuread_application.client.application_id}"
}

resource "azurerm_azuread_service_principal_password" "sp" {
  service_principal_id = "${azurerm_azuread_service_principal.sp.id}"
  value                = "password1"
  end_date             = "2025-01-01T01:02:03Z"
}

resource "azurerm_azuread_service_principal_password" "server" {
  service_principal_id = "${azurerm_azuread_service_principal.server.id}"
  value                = "password2"
  end_date             = "2025-01-01T01:02:03Z"
}

# data "azurerm_subscription" "primary" {}

resource "azurerm_role_assignment" "ra" {
  # scope                = "${data.azurerm_subscription.primary.id}"
  scope                = "${azurerm_container_registry.acr.id}"
  role_definition_name = "Reader"
  principal_id         = "${azurerm_azuread_service_principal.sp.id}"
}

# cluster
resource "azurerm_kubernetes_cluster" "aks" {
  name                = "${var.cluster_name}"
  location            = "${azurerm_resource_group.rg.location}"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  dns_prefix          = "${var.dns_prefix}"

  agent_pool_profile {
    name    = "myapp"
    count   = "${var.agent_count}"
    vm_size = "Standard_B2s"
    os_type = "Linux"
  }

  addon_profile {
    oms_agent {
      enabled                    = true
      log_analytics_workspace_id = "${azurerm_log_analytics_workspace.la_workspace.id}"
    }
  }

  service_principal {
    client_id     = "${azurerm_azuread_application.sp.application_id}"
    client_secret = "password1"
  }

  role_based_access_control {
    azure_active_directory {
      client_app_id     = "${azurerm_azuread_application.client.application_id}"
      server_app_id     = "${azurerm_azuread_application.server.application_id}"
      server_app_secret = "password2"
    }
  }

  tags {
    Environment = "Production"
  }
}

provider "kubernetes" {
  host                   = "${azurerm_kubernetes_cluster.aks.kube_config.0.host}"
  username               = "${azurerm_kubernetes_cluster.aks.kube_config.0.username}"
  password               = "${azurerm_kubernetes_cluster.aks.kube_config.0.password}"
  client_certificate     = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)}"
  client_key             = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)}"
  cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)}"
}

resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "kube-system"
  }
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "tiller"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }

  subject {
    api_group = ""
    kind      = "ServiceAccount"
    name      = "tiller"
    namespace = "kube-system"
  }
}

provider "helm" {
  kubernetes {
    host                   = "${azurerm_kubernetes_cluster.aks.kube_config.0.host}"
    username               = "${azurerm_kubernetes_cluster.aks.kube_config.0.username}"
    password               = "${azurerm_kubernetes_cluster.aks.kube_config.0.password}"
    client_certificate     = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate)}"
    client_key             = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.client_key)}"
    cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.aks.kube_config.0.cluster_ca_certificate)}"
  }
}

resource "helm_release" "ingress" {
  name  = "myapp-nginx-ingress"
  chart = "stable/nginx-ingress"

  set = [
    {
      name  = "rbac.create"
      value = true
    },
  ]
}
jpetitte commented 5 years ago

I figure I will just keep you updated on progress. I've updated the config (it seems like an update to the Azurerm provider went out), and I now have the following config with the following error: * helm_release.ingress: rpc error: code = Unknown desc = configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"

This looks like progress. The config is: 


# Configure the Azure Provider
provider "azurerm" {}

resource "azurerm_resource_group" "rg" {
  name     = "${var.resource_group_name}"
  location = "${var.location}"

  # tags = {
  #   workspace = "myapp"
  # }
}

resource "azurerm_log_analytics_workspace" "la_workspace" {
  name                = "${var.log_analytics_workspace_name}"
  location            = "${var.log_analytics_workspace_location}"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  sku                 = "${var.log_analytics_workspace_sku}"
}

resource "azurerm_log_analytics_solution" "la_solution" {
  solution_name         = "ContainerInsights"
  location              = "${azurerm_log_analytics_workspace.la_workspace.location}"
  resource_group_name   = "${azurerm_resource_group.rg.name}"
  workspace_resource_id = "${azurerm_log_analytics_workspace.la_workspace.id}"
  workspace_name        = "${azurerm_log_analytics_workspace.la_workspace.name}"

  plan {
    publisher = "Microsoft"
    product   = "OMSGallery/ContainerInsights"
  }
}

# acr
resource "azurerm_storage_account" "storage" {
  name                     = "myappacrstorage"
  resource_group_name      = "${azurerm_resource_group.rg.name}"
  location                 = "${azurerm_resource_group.rg.location}"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

resource "azurerm_container_registry" "acr" {
  name                = "myappacr"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  location            = "${azurerm_resource_group.rg.location}"

  # admin_enabled       = true
  sku                = "Classic"
  storage_account_id = "${azurerm_storage_account.storage.id}"
}

# rbac
resource "azurerm_azuread_application" "sp" {
  name                       = "myapp-sp"
  homepage                   = "https://myapp-sp"
  identifier_uris            = ["https://myapp-sp"]
  reply_urls                 = ["https://myapp-sp"]
  available_to_other_tenants = false
  oauth2_allow_implicit_flow = false
}

resource "azurerm_azuread_application" "server" {
  name                       = "myapp-server"
  homepage                   = "https://myapp-server"
  identifier_uris            = ["https://myapp-server"]
  reply_urls                 = ["https://myapp-server"]
  available_to_other_tenants = false
  oauth2_allow_implicit_flow = false
}

resource "azurerm_azuread_application" "client" {
  name                       = "myapp-client"
  homepage                   = "https://myapp-client"
  identifier_uris            = ["https://myapp-client"]
  reply_urls                 = ["https://myapp-client"]
  available_to_other_tenants = false
  oauth2_allow_implicit_flow = false
}

resource "azurerm_azuread_service_principal" "sp" {
  application_id = "${azurerm_azuread_application.sp.application_id}"
}

resource "azurerm_azuread_service_principal" "server" {
  application_id = "${azurerm_azuread_application.server.application_id}"
}

resource "azurerm_azuread_service_principal" "client" {
  application_id = "${azurerm_azuread_application.client.application_id}"
}

resource "azurerm_azuread_service_principal_password" "sp" {
  service_principal_id = "${azurerm_azuread_service_principal.sp.id}"
  value                = "password1"
  end_date             = "2025-01-01T01:02:03Z"
}

resource "azurerm_azuread_service_principal_password" "server" {
  service_principal_id = "${azurerm_azuread_service_principal.server.id}"
  value                = "password2"
  end_date             = "2025-01-01T01:02:03Z"
}

# data "azurerm_subscription" "primary" {}

resource "azurerm_role_assignment" "ra" {
  # scope                = "${data.azurerm_subscription.primary.id}"
  scope                = "${azurerm_container_registry.acr.id}"
  role_definition_name = "Reader"
  principal_id         = "${azurerm_azuread_service_principal.sp.id}"
}

# cluster
resource "azurerm_kubernetes_cluster" "aks" {
  name                = "${var.cluster_name}"
  location            = "${azurerm_resource_group.rg.location}"
  resource_group_name = "${azurerm_resource_group.rg.name}"
  dns_prefix          = "${var.dns_prefix}"

  agent_pool_profile {
    name    = "myapp"
    count   = "${var.agent_count}"
    vm_size = "Standard_B2s"
    os_type = "Linux"
  }

  addon_profile {
    oms_agent {
      enabled                    = true
      log_analytics_workspace_id = "${azurerm_log_analytics_workspace.la_workspace.id}"
    }
  }

  service_principal {
    client_id     = "${azurerm_azuread_application.sp.application_id}"
    client_secret = "password1"
  }

  role_based_access_control {
    azure_active_directory {
      client_app_id     = "${azurerm_azuread_application.client.application_id}"
      server_app_id     = "${azurerm_azuread_application.server.application_id}"
      server_app_secret = "password2"
    }
#########UPDATED#########
    enabled = true
  }

  tags {
    Environment = "Production"
  }
}

#######UPDATED#######
provider "kubernetes" {
  host                   = "${azurerm_kubernetes_cluster.aks.kube_admin_config.0.host}"
  username               = "${azurerm_kubernetes_cluster.aks.kube_admin_config.0.username}"
  password               = "${azurerm_kubernetes_cluster.aks.kube_admin_config.0.password}"
  client_certificate     = "${base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_certificate)}"
  client_key             = "${base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_key)}"
  cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.cluster_ca_certificate)}"
}

resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "kube-system"
  }
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "tiller"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }

  subject {
    api_group = ""
    kind      = "ServiceAccount"
    name      = "tiller"
    namespace = "kube-system"
  }
}

########UPDATED########
provider "helm" {
  kubernetes {
    host                   = "${azurerm_kubernetes_cluster.aks.kube_admin_config.0.host}"
    username               = "${azurerm_kubernetes_cluster.aks.kube_admin_config.0.username}"
    password               = "${azurerm_kubernetes_cluster.aks.kube_admin_config.0.password}"
    client_certificate     = "${base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_certificate)}"
    client_key             = "${base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.client_key)}"
    cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.aks.kube_admin_config.0.cluster_ca_certificate)}"
  }
}

resource "helm_release" "ingress" {
  name  = "myapp-nginx-ingress"
  chart = "stable/nginx-ingress"

  set = [
    {
      name  = "rbac.create"
      value = true
    },
  ]
}
jpetitte commented 5 years ago

Another update. Adding service_account = "tiller" to the "helm" provider eliminated the last error. As I'm now getting no more errors, this looks promising.

TomArcherMsft commented 5 years ago

@jpetitte Thanks for the feedback on this. I'm engaging with the product group to discuss.

TomArcherMsft commented 5 years ago

Adding @grayzu for input.

grayzu commented 5 years ago

@jpetitte,

Thanks for the detailed configuration information and updates as you made progress!

It sounds to me like you have your example fully functional at this, point. Is that correct?

As far as the documentation goes, we will definitely put together a more complete example of using Terraform to deploy AKS using RBAC because as you have found a deployment really requires much more than the azurerm provider in order to enable a real world deployment.

Thanks, MarkG

jpetitte commented 5 years ago

Hey @grayzu

I actually ended up not using terraform because of the struggles (maybe it's just me). For the next project, I'm going back to terraform again with fresh eyes, and it seems to be helping. However, I haven't fully implemented it yet, so we'll see how it goes when I get there.

A full example for deploying AKS with RBAC using active directory via terraform would be awesome.

Also, response time to this issue was a little slow, but you still responded. This type of community help on these issues is a big reason why I choose Azure over other platforms. Keep it up!

TomArcherMsft commented 5 years ago

@jpetitte Thank you for your feedback. Unfortunately, I was on vacation when you originally opened the issue, which caused most of the delay. That is not to make an excuse as we should have better measures in place to respond to customer feedback in a more timely fashion when someone is out. I'm currently looking at ways to improve that aspect of our customer engagement.

As for your kind remarks regarding the community help we provide, that is definitely appreciated by the documentation and feature teams.

Please feel free to reach out again with your feedback!

@MicahMcKittrick-MSFT #please-close