Azure policy addon information is not added when adding another addon

piotrgwiazda commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.2.4

AzureRM Provider Version

3.16.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

Any terraform file containing:


resource "azurerem_kubernetes_cluster" "this" {
  ...
  oms_agent {
    log_analytics_workspace_id = var.workspace_id
  }

  azure_policy_enabled = true

  # First create the cluster without the `key_vault_secrets_provider` section. Then add this section and try apply again.
  key_vault_secrets_provider {
    secret_rotation_enabled = true
    secret_rotation_interval = "2m"
  }
  ...
}

Debug Output/Panic Output

The addonProfiles.azurepolicy section in the JSON sent to Azure API via PUT is missing.

    "addonProfiles": {
      "azureKeyvaultSecretsProvider": {
        "config": {
          "enableSecretRotation": "true",
          "rotationPollInterval": "2m"
        },
        "enabled": true
      },
      "omsagent": {
        "config": {
          "logAnalyticsWorkspaceResourceID": "xxxx"
        },
        "enabled": true
      }
    },

Expected Behaviour

I would expect that every time when I update the cluster, the addonProfiles.azurePolicy section of the JSON sent to Azure API via PUT request is added when the azure_policy_enabled attribute is set to true.

Expecting:

    "addonProfiles": {
      "azurepolicy": {
        "enabled": true
      },
      "azureKeyvaultSecretsProvider": {
        "config": {
          "enableSecretRotation": "true",
          "rotationPollInterval": "2m"
        },
        "enabled": true
      },
      "omsagent": {
        "config": {
          "logAnalyticsWorkspaceResourceID": "xxxx"
        },
        "enabled": true
      }
    }

Actual Behaviour

The addonProfiles.azurePolicy section is added only when the cluster is created, but not on updates. When adding another addon (e.g. key_vault_secrets_provider ), the azure policy addon is removed.

This causes a deployment failure in my environment where I want to implement a custom Azure policy for organization that will Deny any kubernetes cluster that does not have the Azure Policy addon enabled. My policy requires that the addonProfiles.azurePolicy.enabled property exists and is set to true. However, if ti was not blocked by the policy it will probably remove the policy addon from the cluster.

The addonProfiles.azurepolicy section in the JSON sent to Azure API via PUT is missing.

    "addonProfiles": {
      "azureKeyvaultSecretsProvider": {
        "config": {
          "enableSecretRotation": "true",
          "rotationPollInterval": "2m"
        },
        "enabled": true
      },
      "omsagent": {
        "config": {
          "logAnalyticsWorkspaceResourceID": "xxxx"
        },
        "enabled": true
      }
    },

This blocks me from upgrading to AzureRM provider 3.x.x as this is inconsistent with 2.x.x that was always sending the property.

Steps to Reproduce

Terraform apply with azure_policy_enabled = true but without the key_vault_secrets_provider block.
Introduce a change by adding key_vault_secrets_provider block.
Terraform apply

Important Factoids

No response

References

https://github.com/hashicorp/terraform-provider-azurerm/blob/dff3c08bb294f4049bd6e7ec7fca2b15820414fa/internal/services/containers/kubernetes_cluster_resource.go#L1337

    if d.HasChange("aci_connector_linux") || d.HasChange("azure_policy_enabled") || d.HasChange("http_application_routing_enabled") || d.HasChange("oms_agent") || d.HasChange("ingress_application_gateway") || d.HasChange("open_service_mesh_enabled") || d.HasChange("key_vault_secrets_provider") {
        updateCluster = true
        addOns := collectKubernetesAddons(d)
        addonProfiles, err := expandKubernetesAddOns(d, addOns, env)
        if err != nil {
            return err
        }
        existing.ManagedClusterProperties.AddonProfiles = *addonProfiles
    }

When any of the listed properties changes, then addonProfiles are recalculated by calling expandKubernetesAddOns. As a result azure policy addon is removed as it has not changed. If nothing changes, then the block is not executed and the azure policy addon is left "as is".

It seems to be implemented in https://github.com/hashicorp/terraform-provider-azurerm/blob/v3.16.0/internal/services/containers/kubernetes_addons.go#L295

The addon profile is added only when anything changed. However, if the recalculation is triggered by another addon, the azure policy addon is omited.

    if ok := d.HasChange("azure_policy_enabled"); ok {
        v := input["azure_policy_enabled"].(bool)
        props := &containerservice.ManagedClusterAddonProfile{
            Enabled: utils.Bool(v),
            Config: map[string]*string{
                "version": utils.String("v2"),
            },
        }
        addonProfiles[azurePolicyKey] = props
    }

piotrgwiazda commented 2 years ago

I've updated the bug report for completeness.

piotrgwiazda commented 2 years ago

I've confirmed that the azure_policy addon is removed also, when we try to add Open Service Mesh.

piotrgwiazda commented 2 years ago

@ms-henglu please take a look at a draft concept https://github.com/hashicorp/terraform-provider-azurerm/pull/18068 This is my first approach to Go, so it might not be 100% correct, but it is worth looking.

stephybun commented 2 years ago

Closing for the moment since the current implementation is deliberate and working as expected, please see this comment over on #18068 for more details.

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-azurerm