Exclude Managed Private Endpoints from ARM Template Export

[joshwright10]

Problem

The ARM Template exported from Data Factory using the Git integration includes the Managed Private Endpoint resources. As these Private Endpoints are environment specific, these should not be promoted to other environments and should be excluded from the ARM template. In the current state, all MPEs are propagated up to UAT and Production, meaning that the UAT and Production Data Factory instances try to create Managed Private Endpoints to the DEV SQL and Cosmos DB Account.

Our Terraform instances are provisioned via Terraform, and Managed Private Endpoints are configured there, with DEV connections to DEV resources (SQL Servers, Storage Accounts, Key Vaults, Cosmos DB etc.), UAT to UAT resources and Production to Production resources.

A stripped-down version of the ARM Template

{
    "name": "[concat(parameters('factoryName'), '/default')]",
    "type": "Microsoft.DataFactory/factories/managedVirtualNetworks",
    "apiVersion": "2018-06-01",
    "properties": {},
    "dependsOn": []
},
{
    "name": "[concat(parameters('factoryName'), '/default/cosmosdb-dev01-server01-CosmosDb-Sql')]",
    "type": "Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints",
    "apiVersion": "2018-06-01",
    "properties": {
        "privateLinkResourceId": "[parameters('cosmosdb-dev01-server01-CosmosDb-Sql_properties_privateLinkResourceId')]",
        "groupId": "[parameters('cosmosdb-dev01-server01-CosmosDb-Sql_properties_groupId')]",
        "fqdns": "[parameters('cosmosdb-dev01-server01-CosmosDb-Sql_properties_fqdns')]"
    },
    "dependsOn": [
        "[concat(variables('factoryId'), '/managedVirtualNetworks/default')]"
    ]
},
{
    "name": "[concat(parameters('factoryName'), '/default/sql-dev01-server01-SqlServer-sqlServer')]",
    "type": "Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints",
    "apiVersion": "2018-06-01",
    "properties": {
        "privateLinkResourceId": "[parameters('sql-dev01-server01-SqlServer-sqlServer_properties_privateLinkResourceId')]",
        "groupId": "[parameters('sql-dev01-server01-SqlServer-sqlServer_properties_groupId')]",
        "fqdns": "[parameters('sql-dev01-server01-SqlServer-sqlServer_properties_fqdns')]"
    },
    "dependsOn": [
        "[concat(variables('factoryId'), '/managedVirtualNetworks/default')]"
    ]
}

We are using this document Sample pre- and post-deployment script to promote changes from development through to production. The workflow looks like this:

• Data Factory deployed via Terraform.
• Managed Private Endpoints created and managed via Terraform with DEV Data Factory connected to DEV resource (databases, storage accounts etc.)
• Changes published to Git repo using Git integration / vsts_configuration. Repo has the following folders:

  DF-DEV01-MYDATAFACTORY01
  ├── globalParameters/
  │   └── DF-DEV01-MYDATAFACTORY01_GlobalParameters.json
  ├── linkedTemplates/
  │   ├── ArmTemplate_0.json
  │   ├── ArmTemplate_master.json
  │   └── ArmTemplateParameters_master.json
  ├── ARMTemplateForFactory.json
  └── ARMTemplateParametersForFactory.json
  azure-pipelines.yaml

• Pipeline with PrePostDeploymentScript.ps1 triggered when ADF publishes to the Git repo to promote changes through to production.

Solution

• Option 1: Data Factory to not export resources of type Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints to the ARM Template.
• Option 2: PrePostDeploymentScript.Ver2.ps1 script to have an option to strip out Microsoft.DataFactory/factories/managedVirtualNetworks/managedPrivateEndpoints resources.