contributor permission recreated

gevraud commented 5 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Greenfield/Brownfield provisioning

greenfield

Terraform Version

1.8.2

Module Version

8.0.0

AzureRM Provider Version

3.101

Affected Resource(s)/Data Source(s)

azurerm_resource_group_template_deployment

Terraform Configuration Files

module "aks" {
  source               = "Azure/aks/azurerm"
  version              = "8.0.0"
  tracing_tags_enabled = true
  tracing_tags_prefix  = "aks_module_"
  tags       = var.tags

  cluster_name                 = var.kubernetes_cluster_name
  resource_group_name          = azurerm_resource_group.this.name
  location                     = azurerm_resource_group.this.location
  kubernetes_version           = "1.29"
  automatic_channel_upgrade    = "patch"
  sku_tier                     = "Standard"
  identity_ids                 = [azurerm_user_assigned_identity.this.id]
  identity_type                = "UserAssigned"
  microsoft_defender_enabled   = true 
  node_resource_group          = "node-${azurerm_resource_group.this.name}"
  node_os_channel_upgrade      = "NodeImage"
  // Monitoring
  log_analytics_solution = {
    id = azurerm_log_analytics_solution.this.id
  }
  log_analytics_workspace_enabled = true
  log_analytics_workspace = {
    id   = azurerm_log_analytics_workspace.this.id
    name = azurerm_log_analytics_workspace.this.name
  }
  msi_auth_for_monitoring_enabled = true
  // RBAC
  rbac_aad                          = true
  rbac_aad_managed                  = true
  role_based_access_control_enabled = true
  rbac_aad_admin_group_object_ids = [
    data.azuread_group.cluster_admins.object_id
  ]

  // Network
  vnet_subnet_id = module.vnet_aks_dev.subnets["${local.subnets_names[0]}"].id
  private_dns_zone_id = "/subscriptions/xxxx/resourceGroups/yyyy/providers/Microsoft.Network/privateDnsZones/aks.xxx.azmk8s.io"
  prefix = "aks-dev" 
  network_contributor_role_assigned_subnet_ids = {
    subnet1 = module.vnet_aks_dev.subnets["${local.subnets_names[0]}"].id
    subnet2 = module.vnet_aks_dev.subnets["${local.subnets_names[1]}"].id
    subnet3 = module.vnet_aks_dev.subnets["${local.subnets_names[2]}"].id
    subnet4 = module.vnet_aks_dev.subnets["${local.subnets_names[3]}"].id
  }
  private_cluster_enabled             = true
  private_cluster_public_fqdn_enabled = false
  azure_policy_enabled      = true
  net_profile_outbound_type = "userDefinedRouting"
  network_plugin            = "azure"
  network_plugin_mode       = "overlay"
  network_policy            = "calico"   
  // Node Pools
  agents_availability_zones = ["1"]
  agents_count              = null // because autoscalling enabled
  agents_pool_name          = "default"
  agents_labels = {
    type = "system"
  }
  enable_auto_scaling = true
  agents_max_count = 10 // for all
  agents_min_count = 1
  agents_max_pods  = 50
  agents_type      = "VirtualMachineScaleSets"
  temporary_name_for_rotation = "defaulttemp"

  node_pools = {
    worker1 = {
      name                = "worker1"
      vm_size             = "Standard_D2s_v3"
      node_count          = 1
      max_count = 10
      vnet_subnet_id      = module.vnet_aks_dev.subnets["${local.subnets_names[1]}"].id
      enable_auto_scaling = true
      os_disk_size_gb     = 128
      os_sku              = "Ubuntu"
      create_before_destroy = true
    }

  }
  maintenance_window = {
    allowed = [
      {
        day   = "Sunday",
        hours = [20, 23]
      },
    ]
    not_allowed = [
      {
        start = "2035-01-01T20:00:00Z",
        end   = "2035-01-01T21:00:00Z"
      },
    ]
  }
  maintenance_window_node_os = {
    frequency  = "Daily"
    interval   = 1
    start_time = "07:00"
    utc_offset = "+01:00"
    duration   = 16
  }

  depends_on = [
    module.vnet_aks_dev
  ]
}

tfvars variables values

rg_name                 = "rg-xxx-001"
kubernetes_cluster_name = "aks-xxx-dev-001"
kubernetes_dns_prefix   = "dev"
dns_servers             = ["x.x.x.x"]

Debug Output/Panic Output

# module.aks.azurerm_role_assignment.network_contributor_on_subnet["subnet1"] must be replaced
-/+ resource "azurerm_role_assignment" "network_contributor_on_subnet" {
      ~ id                                     = "/subscriptions/xxxxb/resourceGroups/rg-aks-dev-001/providers/Microsoft.Network/virtualNetworks/vnet-aks-dev-001/subnets/snet-aks-dev-001/providers/Microsoft.Authorization/roleAssignments/zzzz" -> (known after apply)
      ~ name                                   = "zzzz" -> (known after apply)
      ~ principal_id                           = "aaaa" # forces replacement -> (known after apply) # forces replacement
      ~ principal_type                         = "ServicePrincipal" -> (known after apply)
      ~ role_definition_id                     = "/subscriptions/xxxx/providers/Microsoft.Authorization/roleDefinitions/aaaa" -> (known after apply)
      + skip_service_principal_aad_check       = (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.aks.azurerm_role_assignment.network_contributor_on_subnet["subnet2"] must be replaced
-/+ resource "azurerm_role_assignment" "network_contributor_on_subnet" {
      ~ id                                     = "/subscriptions/4xxx/resourceGroups/rg-aks-dev-001/providers/Microsoft.Network/virtualNetworks/vnet-aks-dev-001/subnets/snet-aks-dev-002/providers/Microsoft.Authorization/roleAssignments/aaaa" -> (known after apply)
      ~ name                                   = "aaaa" -> (known after apply)
      ~ principal_id                           = "bbbb" # forces replacement -> (known after apply) # forces replacement
      ~ principal_type                         = "ServicePrincipal" -> (known after apply)
      ~ role_definition_id                     = "/subscriptions/xxxx/providers/Microsoft.Authorization/roleDefinitions/aaaa" -> (known after apply)
      + skip_service_principal_aad_check       = (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.aks.azurerm_role_assignment.network_contributor_on_subnet["subnet3"] must be replaced
-/+ resource "azurerm_role_assignment" "network_contributor_on_subnet" {
      ~ id                                     = "/subscriptions/xxxx/resourceGroups/rg-aks-dev-001/providers/Microsoft.Network/virtualNetworks/vnet-aks-dev-001/subnets/snet-aks-dev-003/providers/Microsoft.Authorization/roleAssignments/aaaa" -> (known after apply)
      ~ name                                   = "aaaa" -> (known after apply)
      ~ principal_id                           = "bbbb" # forces replacement -> (known after apply) # forces replacement
      ~ principal_type                         = "ServicePrincipal" -> (known after apply)
      ~ role_definition_id                     = "/subscriptions/xxxx/providers/Microsoft.Authorization/roleDefinitions/aaaa" -> (known after apply)
      + skip_service_principal_aad_check       = (known after apply)
        # (6 unchanged attributes hidden)
    }

  # module.aks.azurerm_role_assignment.network_contributor_on_subnet["subnet4"] must be replaced
-/+ resource "azurerm_role_assignment" "network_contributor_on_subnet" {
      ~ id                                     = "/subscriptions/xxxx/resourceGroups/rg-aks-dev-001/providers/Microsoft.Network/virtualNetworks/vnet-aks-dev-001/subnets/snet-aks-dev-004/providers/Microsoft.Authorization/roleAssignments/aaaa" -> (known after apply)
      ~ name                                   = "aaaa" -> (known after apply)
      ~ principal_id                           = "bbbb" # forces replacement -> (known after apply) # forces replacement
      ~ principal_type                         = "ServicePrincipal" -> (known after apply)
      ~ role_definition_id                     = "/subscriptions/xxxx/providers/Microsoft.Authorization/roleDefinitions/aaaa" -> (known after apply)
      + skip_service_principal_aad_check       = (known after apply)
        # (6 unchanged attributes hidden)
    }

Expected Behaviour

no recreation should happen

Actual Behaviour

it recreates the permission each time Terraform is running

Steps to Reproduce

No response

Important Factoids

No response

References

No response

oscarwest commented 3 months ago

We get the same thing every time we plan/apply.

module.aks.azurerm_role_assignment.application_gateway_resource_group_reader and module.aks.azurerm_role_assignment.application_gateway_existing_vnet_network_contributor

will be re-created every time (known after apply)

zioproto commented 3 months ago

For some reason the value of the principal_id is changing at every Terraform plan.

https://github.com/Azure/terraform-azurerm-aks/blob/4729aee8c064da747186a14d6c3575c67e673692/role_assignments.tf#L42

@gevraud in your actual plan what do you see instead of "bbbb" ? Does the value actually change?

zioproto commented 3 months ago

I confirm I was able to reproduce the problem on 8.0.0 but not on 9.0.0 or 9.1.0.

zioproto commented 3 months ago

It is working properly in 9.0.0 and newer versions because since the version 9.0.0 there is this change: 6abee9eaf84ef95f7ca59017ff5251dea7957561 from PR https://github.com/Azure/terraform-azurerm-aks/pull/554

AKS had a change in the default behaviour that caused a state drift. When using 8.0.0 you should have also the following on top of the proposed plan:

Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:

  # module.aks.azurerm_kubernetes_cluster.main has changed
  ~ resource "azurerm_kubernetes_cluster" "main" {
        id                                  = "/subscriptions/xxxx/resourceGroups/<rg>/providers/Microsoft.ContainerService/managedClusters/ccccc"
        name                                = "cccccc"
        # (39 unchanged attributes hidden)

      ~ identity {
          + identity_ids = []
            # (3 unchanged attributes hidden)
        }

        # (7 unchanged blocks hidden)
    }

Those identity ids change propagate into the problem described in this issue.

Could you please try to reproduce on version 9.1.0 and confirm the problem is fixed ?

thanks

gevraud commented 3 months ago

For some reason the value of the principal_id is changing at every Terraform plan.

https://github.com/Azure/terraform-azurerm-aks/blob/4729aee8c064da747186a14d6c3575c67e673692/role_assignments.tf#L42

@gevraud in your actual plan what do you see instead of "bbbb" ? Does the value actually change? Hello,

I created permission without the module. I don't use the module permission anymore.

Regards

Azure / terraform-azurerm-aks