microsoft_defender for `azure_kubernetes_service` can not be disabled

[rvashishth]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.1.9

AzureRM Provider Version

3.14.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

NA

Debug Output/Panic Output

NA

Expected Behaviour

Azurerm should not create log analytics workspace in default resource group.

if azurerm_kubernetes_cluster is created without specifiying any microsoft_defender block it should not create any security profile/azure defender.

az aks show -g resource_group_name -n aks_cluster_name should have following result

  "securityProfile": {
    "azureDefender": null
  },

Actual Behaviour

Even if microsoft_defender block is not present on azurerm_kubernetes_cluster resource, azurerm creates a log analytics workspace in default resource group and assign it to aks cluster.

Earlier we have used azurerm 3.0.0 which did not create any log analytics workspace.

Steps to Reproduce

Create azurerm_kubernetes_cluster resource using azurerm version 3.14.0 without defining any microsoft_defender block

terraform apply

get azure aks cluster will result log analytics workspace connection with aks. az aks show -g resource_group_name -n aks_cluster_name will have following block in it.

  "securityProfile": {
    "azureDefender": {
      "enabled": true,
      "logAnalyticsWorkspaceResourceId": "/subscriptions/34es54w-f86e-443b-9735-17679033a4e6/resourcegroups/DefaultResourceGroup-CUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-4gsr4s44-f86e-443b-9735-17679033a4e6-CUS"
    }
  },

Important Factoids

No response

References

https://github.com/hashicorp/terraform-provider-azurerm/pull/16218 https://github.com/hashicorp/terraform-provider-azurerm/issues/15063

[rvashishth]

analytics workspace got created in default resource group even when microsoft_defender is not used in aks resource. it result in aks resource update and re-creation of role assignments wherever this aks resource is referenced.

terraform plan results in following, while there should not be any change in infra since microsoft_defender wasn't provided while creating aks cluster.

      - microsoft_defender {
          - log_analytics_workspace_id = "/subscriptions/****/resourcegroups/DefaultResourceGroup-CUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-****-CUS" -> null
        }

[ms-henglu]

Hi @rvashishth ,

Was this cluster created by terraform azurerm provider? Or was it created by Portal and then imported into terraform?

[rvashishth]

Hi @rvashishth ,

Was this cluster created by terraform azurerm provider? Or was it created by Portal and then imported into terraform?

using terraform azurerm provider.

[rvashishth]

I worked with azure team, and it seems this issue is with terraform provider only. using azure cli/portal we can create aks by disabling defender, without creating any integration with workspace resource id.

[gzur]

I worked with azure team, and it seems this issue is with terraform provider only. using azure cli/portal we can create aks by disabling defender, without creating any integration with workspace resource id.

Did this result in anything actionable (another issue/PR or something)?

I am currently battling this, and using the azure cli/portal is not really an option.

Furthermore, I would be willing to take a look at fixing this, but I have no idea where to start.

[bitsofinfo]

please fix this! I can't update a cluster anymore where this statement previously existed.

[bitsofinfo]

@rvashishth question:

I constantly get the error:

│ Error: updating Cluster: (Managed Cluster Name "xxxxx" / Resource Group "xxxxxx"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="AzureDefenderFeatureFlagNotEnabled" Message="AzureDefender installation is not allowed since feature 'Microsoft.ContainerService/AKS-AzureDefender' is not enabled. Please see https://aka.ms/aks/previews for how to enable features."

how did you enable this? I can't find anywhere on how to enable this preview feature to begin with, the link is of no help in the error message

[gzur]

Just so I understand you correctly, is your flow somehow like this?

1. terraform apply works for creating a new cluster
2. Subsequent attempts at terraform apply fail with the aforementioned error?

Are you using the terraform-azurerm-aks module - and if you are, have you tried the latest version?

Do you have auto provisioning for Microsoft Defender for Cloud enabled in the portal?

[gzur]

I have the sneaking suspicion that this StatusCode=400 is the result of something like the following scenario (which is entirely hypothetical based on black box debugging):

1. terraform apply creates a cluster
2. Defender auto-provisions itself into the cluster after a few minutes.
3. Subsequent terraform apply commands detect this and try to remove Defender, since it doesn't match the terraform state.
4. Something in the backend doesn't like this and maybe returns a 403 - since whatever permissions you are giving terraform do not trump whatever permissions the automatic provisioner has.
5. Some other code path - left from since Defender was in "Preview" mode - assumes all 4xx errors are due to Defender not being Enabled and wrongly returns the aforementioned error message.
6. Much sadness

[grodr]

@rvashishth question:

I constantly get the error:

│ Error: updating Cluster: (Managed Cluster Name "xxxxx" / Resource Group "xxxxxx"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="AzureDefenderFeatureFlagNotEnabled" Message="AzureDefender installation is not allowed since feature 'Microsoft.ContainerService/AKS-AzureDefender' is not enabled. Please see https://aka.ms/aks/previews for how to enable features."

how did you enable this? I can't find anywhere on how to enable this preview feature to begin with, the link is of no help in the error message

I am hitting this same error. I am able to successfully create an aks cluster with an initial terraform apply, but subsequent terraform applys fail with

... containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="AzureDefenderFeatureFlagNotEnabled" Message="AzureDefender installation is not allowed since feature 'Microsoft.ContainerService/AKS-AzureDefender' is not enabled. Please see https://aka.ms/aks/previews for how to enable features."

I do have Auto provision on for Azure Policy for Kubernetes. Adding ignore_changes for microsoft_defender alleviates some of these problems as terraform no longer tries to remove the defender settting, but applying in place changes to the cluster still cause this error.

[PolinaApostolovaCOT]

We managed to solve this on our end (and keep Defender for Cloud) by registering the Defender for Cloud feature, which seems to be different from the auto provisioning.

If you run either of these in the subscription where the AKS cluster is:

az feature registration show --provider-namespace "Microsoft.ContainerService" --name "AKS-AzureDefender"

OR

az feature show --namespace "Microsoft.ContainerService" --name "AKS-AzureDefender"

And you will see state: NotRegistered

To register it you have to run az feature register --namespace "Microsoft.ContainerService" --name "AKS-AzureDefender"

Be aware it might take a while. This command is useful if you want to check the current state:

az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/AKS-AzureDefender')].{Name:name,State:properties.state}"

I am not sure what the deal with this is and why the feature has to be registered (as the Terraform Provider error indicates), may be because it was in preview only until recently? We've been trying to upgrade the cluster all week and Defender has been getting in the way, which is the exact opposite of what a security feature should be doing!

I hope this helps you guys!

[grodr]

@PolinaApostolovaCOT This worked for me, thank you!

[man0s]

@PolinaApostolovaCOT That worked for us as-well! Thanks 💯

[dkuzmenok]

Shouldn't this do the same in native way:

resource "azurerm_resource_provider_registration" "example" {
  name = "Microsoft.ContainerService"

  feature {
    name       = "AKS-AzureDefender"
    registered = true
  }
}

[tspearconquest]

It appears to me that Microsoft has forcibly enabled this at the subscription level when clusters are onboarded to Azure Security Center.

Our existing cluster got this enabled in mid-december and it has broken our Gatekeeper instance.

In order to fix it, I needed to add an Azure Policy Exemption. I did this via the portal and then imported it to Terraform:

data "azurerm_policy_definition" "deploy_aks_azure_policy_addon" {
  display_name = "Deploy Azure Policy Add-on to Azure Kubernetes Service clusters"
}

data "azurerm_policy_assignment" "defender_for_containers_provisioning_azure_policy_addon_for_kubernetes" {
  name     = "Defender for Containers provisioning Azure Policy Addon for Kub" ## This field has a maximum length of 64 characters
  scope_id = "/subscriptions/${local.subscription_id}"
}

resource "azurerm_subscription_policy_exemption" "prod_subscription_exemption" {
  name = "1e3fb72924f744d9b306ca59" ## appears to be randomly assigned by the portal when creating the exemption

  display_name = "Exemption for Defender for Containers provisioning Azure Policy Addon for Kubernetes"
  description  = "Gatekeeper" ## Azure Policy uses Gatekeeper but our Gatekeeper is customized, so pick your own description here

  subscription_id      = "/subscriptions/${local.subscription_id}"

  exemption_category   = "Mitigated"
  policy_assignment_id = data.azurerm_policy_assignment.defender_for_containers_provisioning_azure_policy_addon_for_kubernetes.id
}

Once I added the above code to Terraform the next terraform plan showed the microsoft_defender block being removed from the cluster:

        }
        # (26 unchanged attributes hidden)

      - microsoft_defender {
          - log_analytics_workspace_id = "/subscriptions/<subscription_id>/resourcegroups/DefaultResourceGroup-EUS2/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-<subscription_id>-EUS2" -> null
        }

        # (8 unchanged blocks hidden)

[articice]

The "native" recipe from @dkuzmenok didn't work for me, but I created this workaround using azapi provider and Features - Register API call:

resource "azapi_resource_action" "defender_registration_containerservice" {
  type = "Microsoft.ContainerService/features/register@2021-07-01"
  resource_id = "/subscriptions/${data.azurerm_client_config.current.subscription_id}/providers/Microsoft.Features/providers/Microsoft.ContainerService/features/AKS-AzureDefender/register"
  response_export_values = ["properties.state"]
  method = "POST"
}

note, type feels wrong due to azapi validation weirdness, e.g. checking registration state would be like this:

data "azapi_resource" "check_defender_registration_containerservice" {
  type = "Microsoft.Features/providers@2021-07-01"
  parent_id = "/subscriptions/${data.azurerm_client_config.current.subscription_id}"
  response_export_values = ["properties.state"]
  name = "Microsoft.ContainerService/features/AKS-AzureDefender"
}

I also added this to cluster declaration, to avoid extra output

  lifecycle {
    ignore_changes = [
      azure_policy_enabled,
      microsoft_defender
    ]
  }