policy_definitions exclusion not working in 21V

[huqianghui]

Versions

terraform: Terraform v1.1.7 on darwin_amd64

azure provider: hashicorp/azurerm v2.99.0

module: source = "Azure/caf-enterprise-scale/azurerm" version = "1.1.4"

Description

In China Azure Cloud, some built-in policies have not deployed yet. When I tried to exclude the policy definitons from terrafrom file as below files. But the built-in policy lookup errors always appear.

error message as below:

Error: reading Policy Definition "18adea5e-f416-4d0f-8aa8-d24321e3e274": policy.DefinitionsClient#GetBuiltIn: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="PolicyDefinitionNotFound" Message="The policy definition '18adea5e-f416-4d0f-8aa8-d24321e3e274' could not be found." │ │ with module.enterprisescale.data.azurermpolicydefinition.externallookup["/providers/Microsoft.Authorization/policyDefinitions/18adea5e-f416-4d0f-8aa8-d24321e3e274"], │ on .terraform/modules/enterprisescale/locals.policyassignments.tf line 157, in data "azurermpolicydefinition" "externallookup": │ 157: data "azurermpolicydefinition" "externallookup" { │

step 1 wirte the tf file as single subscription file' s instruction. librarypath = "${path.root}/lib" deploycorelandingzones = true step 2 add exclusion file modify main.tf : librarypath = "${path.root}/lib" deploycorelandingzones = true step 3 run terrafrom init & terrafrom plan

Screenshots

Additional context /single-subscription/lib/archetypeexclusioneslandingzones.tmpl.json

{ "excludeeslandingzones": { "policyassignments": [ "Deploy-MDFC-Config", "Deploy-LX-Arc-Monitoring", "Deploy-VM-Monitoring", "Deploy-VMSS-Monitoring", "Deploy-WS-Arc-Monitoring", "Deploy-AzActivity-Log", "Deploy-Resource-Diag", "Deploy-Log-Analytics", "Deny-IP-Forwarding", "Deny-RDP-From-Internet", "Deny-Storage-http", "Deny-Subnet-Without-Nsg", "Deploy-AKS-Policy", "Deploy-SQL-DB-Auditing", "Deploy-VM-Backup", "Deploy-SQL-Security", "Deny-Priv-Escalation-AKS", "Deny-Priv-Containers-AKS", "Deny-http-Ingress-AKS" ], "policydefinitions": [ "Append-AppService-httpsonly", "Append-AppService-latestTLS", "Append-KV-SoftDelete", "Append-Redis-disableNonSslPort", "Append-Redis-sslEnforcement", "Audit-MachineLearning-PrivateEndpointId", "Deny-AA-child-resources", "Deny-AppGW-Without-WAF", "Deny-AppServiceApiApp-http", "Deny-AppServiceFunctionApp-http", "Deny-AppServiceWebApp-http", "Deny-Databricks-NoPublicIp", "Deny-Databricks-Sku", "Deny-Databricks-VirtualNetwork", "Deny-MachineLearning-Aks", "Deny-MachineLearning-Compute-SubnetId", "Deny-MachineLearning-Compute-VmSize", "Deny-MachineLearning-ComputeCluster-RemoteLoginPortPublicAccess", "Deny-MachineLearning-ComputeCluster-Scale", "Deny-MachineLearning-HbiWorkspace", "Deny-MachineLearning-PublicAccessWhenBehindVnet", "Deny-MachineLearning-PublicNetworkAccess", "Deny-MySql-http", "Deny-PostgreSql-http", "Deny-Private-DNS-Zones", "Deny-PublicEndpoint-MariaDB", "Deny-PublicIP", "Deny-RDP-From-Internet", "Deny-Redis-http", "Deny-Sql-minTLS", "Deny-SqlMi-minTLS", "Deny-Storage-minTLS", "Deny-Subnet-Without-Nsg", "Deny-Subnet-Without-Udr", "Deny-VNET-Peer-Cross-Sub", "Deny-VNet-Peering", "Deploy-ASC-SecurityContacts", "Deploy-Budget", "Deploy-Custom-Route-Table", "Deploy-DDoSProtection", "Deploy-Diagnostics-AA", "Deploy-Diagnostics-ACI", "Deploy-Diagnostics-ACR", "Deploy-Diagnostics-AnalysisService", "Deploy-Diagnostics-ApiForFHIR", "Deploy-Diagnostics-APIMgmt", "Deploy-Diagnostics-ApplicationGateway", "Deploy-Diagnostics-CDNEndpoints", "Deploy-Diagnostics-CognitiveServices", "Deploy-Diagnostics-CosmosDB", "Deploy-Diagnostics-Databricks", "Deploy-Diagnostics-DataExplorerCluster", "Deploy-Diagnostics-DataFactory", "Deploy-Diagnostics-DLAnalytics", "Deploy-Diagnostics-EventGridSub", "Deploy-Diagnostics-EventGridSystemTopic", "Deploy-Diagnostics-EventGridTopic", "Deploy-Diagnostics-ExpressRoute", "Deploy-Diagnostics-Firewall", "Deploy-Diagnostics-FrontDoor", "Deploy-Diagnostics-Function", "Deploy-Diagnostics-HDInsight", "Deploy-Diagnostics-iotHub", "Deploy-Diagnostics-LoadBalancer", "Deploy-Diagnostics-LogicAppsISE", "Deploy-Diagnostics-MariaDB", "Deploy-Diagnostics-MediaService", "Deploy-Diagnostics-MlWorkspace", "Deploy-Diagnostics-MySQL", "Deploy-Diagnostics-NetworkSecurityGroups", "Deploy-Diagnostics-NIC", "Deploy-Diagnostics-PostgreSQL", "Deploy-Diagnostics-PowerBIEmbedded", "Deploy-Diagnostics-RedisCache", "Deploy-Diagnostics-Relay", "Deploy-Diagnostics-SignalR", "Deploy-Diagnostics-SQLElasticPools", "Deploy-Diagnostics-SQLMI", "Deploy-Diagnostics-TimeSeriesInsights", "Deploy-Diagnostics-TrafficManager", "Deploy-Diagnostics-VirtualNetwork", "Deploy-Diagnostics-VM", "Deploy-Diagnostics-VMSS", "Deploy-Diagnostics-VNetGW", "Deploy-Diagnostics-WebServerFarm", "Deploy-Diagnostics-Website", "Deploy-Diagnostics-WVDAppGroup", "Deploy-Diagnostics-WVDHostPools", "Deploy-Diagnostics-WVDWorkspace", "Deploy-FirewallPolicy", "Deploy-MySQL-sslEnforcement", "Deploy-Nsg-FlowLogs-to-LA", "Deploy-Nsg-FlowLogs", "Deploy-PostgreSQL-sslEnforcement", "Deploy-Sql-AuditingSettings", "Deploy-SQL-minTLS", "Deploy-Sql-SecurityAlertPolicies", "Deploy-Sql-Tde", "Deploy-Sql-vulnerabilityAssessments", "Deploy-SqlMi-minTLS", "Deploy-Storage-sslEnforcement", "Deploy-VNET-HubSpoke", "Deploy-Windows-DomainJoin" ], "policysetdefinitions": [ "Deny-PublicPaaSEndpoints", "Deploy-ASCDF-Config", "Deploy-Diagnostics-LogAnalytics", "Deploy-Private-DNS-Zones", "Deploy-Sql-Security", "Enforce-Encryption-CMK", "Enforce-EncryptTransit" ], "roledefinitions": [], "archetypeconfig": { "parameters": {}, "accesscontrol": {} } } }

[krowlandson]

Great question, and thank you for logging this @huqianghui

The exclusion capability uses a filter on a for_each loop, and only works at the scope where an item is specified.

As such, you may need to create multiple exclusion_ files to get this working for Azure China.

This is an area I am keen to incorporate natively within the module so would be happy to work with you to update the module to support this if you are interested?

To make this a bit quicker and easier for you in the meantime, you can refer to the following exclusion definitions which will help you to effectively filter out ALL items from ALL archetypes:

archetype_exclusion_es_connectivity.json

{
    "exclude_es_connectivity": {
        "policy_assignments": [
            "Enable-DDoS-VNET"
        ],
        "policy_definitions": [],
        "policy_set_definitions": [],
        "role_definitions": [],
        "archetype_config": {
            "parameters": {},
            "access_control": {}
        }
    }
}

archetype_exclusion_es_corp.json

{
    "exclude_es_corp": {
        "policy_assignments": [
            "Deny-Public-Endpoints",
            "Deploy-Private-DNS-Zones",
            "Deny-DataB-Pip",
            "Deny-DataB-Sku",
            "Deny-DataB-Vnet"
        ],
        "policy_definitions": [],
        "policy_set_definitions": [],
        "role_definitions": [],
        "archetype_config": {
            "parameters": {},
            "access_control": {}
        }
    }
}

archetype_exclusion_es_identity.json

{
    "exclude_es_identity": {
        "policy_assignments": [
            "Deny-Public-IP",
            "Deny-RDP-From-Internet",
            "Deny-Subnet-Without-Nsg",
            "Deploy-VM-Backup"
        ],
        "policy_definitions": [],
        "policy_set_definitions": [],
        "role_definitions": [],
        "archetype_config": {
            "parameters": {},
            "access_control": {}
        }
    }
}

archetype_exclusion_es_landing_zones.json

{
    "exclude_es_landing_zones": {
        "policy_assignments": [
            "Deny-IP-Forwarding",
            "Deny-RDP-From-Internet",
            "Deny-Storage-http",
            "Deny-Subnet-Without-Nsg",
            "Deploy-AKS-Policy",
            "Deploy-SQL-DB-Auditing",
            "Deploy-SQL-Threat",
            "Deploy-VM-Backup",
            "Deny-Priv-Escalation-AKS",
            "Deny-Priv-Containers-AKS",
            "Enable-DDoS-VNET",
            "Enforce-AKS-HTTPS",
            "Enforce-TLS-SSL"
        ],
        "policy_definitions": [],
        "policy_set_definitions": [],
        "role_definitions": [],
        "archetype_config": {
            "parameters": {},
            "access_control": {}
        }
    }
}

archetype_exclusion_es_management.json

{
    "exclude_es_management": {
        "policy_assignments": [
            "Deploy-Log-Analytics"
        ],
        "policy_definitions": [],
        "policy_set_definitions": [],
        "role_definitions": [],
        "archetype_config": {
            "parameters": {},
            "access_control": {}
        }
    }
}

archetype_exclusion_es_root.json

{
  "exclude_es_root": {
    "policy_assignments": [
      "Deploy-ASC-Monitoring",
      "Deploy-MDFC-Config",
      "Deploy-AzActivity-Log",
      "Deploy-LX-Arc-Monitoring",
      "Deploy-Resource-Diag",
      "Deploy-VM-Monitoring",
      "Deploy-VMSS-Monitoring",
      "Deploy-WS-Arc-Monitoring"
    ],
    "policy_definitions": [
      "Append-AppService-httpsonly",
      "Append-AppService-latestTLS",
      "Append-KV-SoftDelete",
      "Append-Redis-disableNonSslPort",
      "Append-Redis-sslEnforcement",
      "Audit-MachineLearning-PrivateEndpointId",
      "Deny-AA-child-resources",
      "Deny-AppGW-Without-WAF",
      "Deny-AppServiceApiApp-http",
      "Deny-AppServiceFunctionApp-http",
      "Deny-AppServiceWebApp-http",
      "Deny-Databricks-NoPublicIp",
      "Deny-Databricks-Sku",
      "Deny-Databricks-VirtualNetwork",
      "Deny-MachineLearning-Aks",
      "Deny-MachineLearning-Compute-SubnetId",
      "Deny-MachineLearning-Compute-VmSize",
      "Deny-MachineLearning-ComputeCluster-RemoteLoginPortPublicAccess",
      "Deny-MachineLearning-ComputeCluster-Scale",
      "Deny-MachineLearning-HbiWorkspace",
      "Deny-MachineLearning-PublicAccessWhenBehindVnet",
      "Deny-MachineLearning-PublicNetworkAccess",
      "Deny-MySql-http",
      "Deny-PostgreSql-http",
      "Deny-Private-DNS-Zones",
      "Deny-PublicEndpoint-MariaDB",
      "Deny-PublicIP",
      "Deny-RDP-From-Internet",
      "Deny-Redis-http",
      "Deny-Sql-minTLS",
      "Deny-SqlMi-minTLS",
      "Deny-Storage-minTLS",
      "Deny-Subnet-Without-Nsg",
      "Deny-Subnet-Without-Udr",
      "Deny-VNET-Peer-Cross-Sub",
      "Deny-VNET-Peering-To-Non-Approved-VNETs",
      "Deny-VNet-Peering",
      "Deploy-ASC-SecurityContacts",
      "Deploy-Budget",
      "Deploy-Custom-Route-Table",
      "Deploy-DDoSProtection",
      "Deploy-Diagnostics-AA",
      "Deploy-Diagnostics-ACI",
      "Deploy-Diagnostics-ACR",
      "Deploy-Diagnostics-AnalysisService",
      "Deploy-Diagnostics-ApiForFHIR",
      "Deploy-Diagnostics-APIMgmt",
      "Deploy-Diagnostics-ApplicationGateway",
      "Deploy-Diagnostics-CDNEndpoints",
      "Deploy-Diagnostics-CognitiveServices",
      "Deploy-Diagnostics-CosmosDB",
      "Deploy-Diagnostics-Databricks",
      "Deploy-Diagnostics-DataExplorerCluster",
      "Deploy-Diagnostics-DataFactory",
      "Deploy-Diagnostics-DLAnalytics",
      "Deploy-Diagnostics-EventGridSub",
      "Deploy-Diagnostics-EventGridSystemTopic",
      "Deploy-Diagnostics-EventGridTopic",
      "Deploy-Diagnostics-ExpressRoute",
      "Deploy-Diagnostics-Firewall",
      "Deploy-Diagnostics-FrontDoor",
      "Deploy-Diagnostics-Function",
      "Deploy-Diagnostics-HDInsight",
      "Deploy-Diagnostics-iotHub",
      "Deploy-Diagnostics-LoadBalancer",
      "Deploy-Diagnostics-LogicAppsISE",
      "Deploy-Diagnostics-MariaDB",
      "Deploy-Diagnostics-MediaService",
      "Deploy-Diagnostics-MlWorkspace",
      "Deploy-Diagnostics-MySQL",
      "Deploy-Diagnostics-NetworkSecurityGroups",
      "Deploy-Diagnostics-NIC",
      "Deploy-Diagnostics-PostgreSQL",
      "Deploy-Diagnostics-PowerBIEmbedded",
      "Deploy-Diagnostics-RedisCache",
      "Deploy-Diagnostics-Relay",
      "Deploy-Diagnostics-SignalR",
      "Deploy-Diagnostics-SQLElasticPools",
      "Deploy-Diagnostics-SQLMI",
      "Deploy-Diagnostics-TimeSeriesInsights",
      "Deploy-Diagnostics-TrafficManager",
      "Deploy-Diagnostics-VirtualNetwork",
      "Deploy-Diagnostics-VM",
      "Deploy-Diagnostics-VMSS",
      "Deploy-Diagnostics-VNetGW",
      "Deploy-Diagnostics-WebServerFarm",
      "Deploy-Diagnostics-Website",
      "Deploy-Diagnostics-WVDAppGroup",
      "Deploy-Diagnostics-WVDHostPools",
      "Deploy-Diagnostics-WVDWorkspace",
      "Deploy-FirewallPolicy",
      "Deploy-MySQL-sslEnforcement",
      "Deploy-Nsg-FlowLogs-to-LA",
      "Deploy-Nsg-FlowLogs",
      "Deploy-PostgreSQL-sslEnforcement",
      "Deploy-Sql-AuditingSettings",
      "Deploy-SQL-minTLS",
      "Deploy-Sql-SecurityAlertPolicies",
      "Deploy-Sql-Tde",
      "Deploy-Sql-vulnerabilityAssessments",
      "Deploy-SqlMi-minTLS",
      "Deploy-Storage-sslEnforcement",
      "Deploy-VNET-HubSpoke",
      "Deploy-Windows-DomainJoin"
    ],
    "policy_set_definitions": [
      "Deny-PublicPaaSEndpoints",
      "Deploy-Diagnostics-LogAnalytics",
      "Deploy-MDFC-Config",
      "Deploy-Private-DNS-Zones",
      "Deploy-Sql-Security",
      "Enforce-Encryption-CMK",
      "Enforce-EncryptTransit"
    ],
    "role_definitions": [
      "Network-Subnet-Contributor",
      "Application-Owners",
      "Network-Management",
      "Security-Operations",
      "Subscription-Owner"
    ],
    "archetype_config": {
      "parameters": {},
      "access_control": {}
    }
  }
}

Currently all other archetypes are equivalent to default_empty which contains no references to policies or roles.

However, I would highly recommend filtering these out one at a time to work out which explicitly need removing. You can then choose how to optimise you configuration based on what needs to be removed.

For example, the Policy Definition you mention above is referenced in the following Policy Set Definition:

https://github.com/Azure/terraform-azurerm-caf-enterprise-scale/blob/0b343c6ba9ce4595a0e494f97d2a439845a1cb2a/modules/archetypes/lib/policy_set_definitions/policy_set_definition_es_enforce_encryption_cmk.tmpl.json#L308

Knowing this, you could choose whether to:

1. Remove this Policy Set Definition and associated assignments(s)
2. Replace this Policy Set Definition with a custom version which removes just the Policy Definitions not supported by Azure China (copy both the Policy Set Definition and Policy Assignment templates into your custom lib folder, and edit them as needed)

But just to re-iterate, I am happy to incorporate this into the module natively with your assistance as I know how we can do this without a breaking change for Azure public customers.

[huqianghui]

Thank you for your quick response and suggestion. @krowlandson I agree with you. Actually I have deleted or modified polies or assignment explicitly and bulit successfully. It's a great pleasure to have the opportunity to work with you. Actually, I have also met some other issues.
Maybe we can fork a new branch and push the changes for azure china cloud or make the configuration more flexible for different cloud enviroment like azure china or gov.

[jtracey93]

Trigger ADO Sync

[krowlandson]

Trigger ADO Sync

[krowlandson]

@lachaves - this GHI is related to the backlog item you are working on for adding support for AzureChinaCloud 👍🏻

[paul-hugill]

In case it is any use, this es_root exclusions worked for me to deploy in AzureChina (we are also leaving off the VNET DDOS assignment anywhere else but that's more just our config I think and we were able to leave the rest in).
Some of the roles have an issue with Microsoft.Support/* it would seem too.

{
    "exclude_es_root": {
        "policy_assignments": [
            "Deploy-MDFC-Config",
            "Deploy-AzActivity-Log",
            "Deploy-LX-Arc-Monitoring",
            "Deploy-WS-Arc-Monitoring"
        ],
        "policy_definitions": [
            "Deploy-Budget"
        ],
        "policy_set_definitions": [
            "Deny-PublicPaaSEndpoints",
            "Deploy-MDFC-Config",
            "Deploy-Private-DNS-Zones",
            "Deploy-Sql-Security",
            "Enforce-Encryption-CMK"
        ],
        "role_definitions": [
            "Network-Subnet-Contributor",
            "Application-Owners",
            "Network-Management",
            "Security-Operations",
            "Subscription-Owner"
        ],
        "archetype_config": {
            "parameters": {},
            "access_control": {}
        }
    }
}

For the policy sets, it's not every policy in them that is the issue but seemed to be a good chunk, these are at least ones that I know of when I first tried and was using v2.0.2 but I didn't try to get the individual ones when I updated to v3.1.2:

2465583e-4e78-4c15-b6be-a36cbc7c8b0f - Configure Azure Activity logs to stream to specified Log Analytics workspace
b7021b2b-08fd-4dc0-9de7-3c6ece09faf9 - Configure Azure Defender for Resource Manager to be enabled
b40e7bcd-a1e5-47fe-b9cf-2f534d0bfb7d - Configure Azure Defender for App Service to be enabled
2370a3c1-4a25-4283-a91a-c9c1a145fb2f - Configure Azure Defender for DNS to be enabled
1f725891-01c0-420a-9059-4fa46cb770b7 - Configure Azure Defender for Key Vaults to be enabled
44433aa3-7ec2-4002-93ea-65c65ff0310a - Configure Azure Defender for open-source relational databases to be enabled
50ea7265-7d8c-429e-9a7d-ca1f410191c3 - Configure Azure Defender for SQL servers on machines to be enabled
74c30959-af11-47b3-9ed2-a26e03f427a3 - Configure Azure Defender for Storage to be enabled
c9299215-ae47-4f50-9c54-8a392f68a052 - Public network access should be disabled for MySQL flexible servers
5e1de0e3-42cb-4ebc-a86d-61d0c619ca48 - Public network access should be disabled for PostgreSQL flexible servers
55615ac9-af46-4a59-874e-391cc3dfb490 - Azure Key Vault should have firewall enabled
9d2b61b4-1d14-4a63-be30-d4498e7ad2cf - Configure Log Analytics extension on Azure Arc enabled Linux servers. See deprecation notice below
69af7d4a-7b18-4044-93a9-2651498ef203 - Configure Log Analytics extension on Azure Arc enabled Windows servers
ac673a9a-f77d-4846-b2d8-a57f8e1c01d4 - [Preview]: Configure Azure Key Vaults to use private DNS zones
0b026355-49cb-467b-8ac4-f777874e175a - Configure Azure Web PubSub Service to use private DNS zones
18adea5e-f416-4d0f-8aa8-d24321e3e274 - PostgreSQL servers should use customer-managed keys to encrypt data at rest
051cba44-2429-45b9-9649-46cec11c7119 - Azure API for FHIR should use a customer-managed key to encrypt data at rest
83cef61d-dbd1-4b20-a4fc-5fbc7da10833 - MySQL servers should use customer-managed keys to encrypt data at rest

I'd be very happy to test out any changes to fully support AzureChina, I am not sure if we will use the Networking/VWAN config from this module yet (in Azure Global we already had the VWAN before deploying this) but everything else would be great.

Depending on where things are sitting timeframe-wise, I might be tempted to come up with the custom policy sets so that we get the policies that are supported but lower down my to do list currently.

[matt-FFFFFF]

Will be documentation

[matt-FFFFFF]

AB#27082

[scdubay]

I was struggling with this issue as well.. It turns out that the exclusions only work if they have already been deployed. Why is this. IT takes a long time to do the deployment and if you want to make changes ahead of time, like excluding all or some of the policy stuff, it would be nice to be able to do it right from the beginning.