Resource not available - Managed Virtual Networks for Data Factory

[clstokes]

Managed Virtual Networks for Data Factory are not available in the Azure Native provider.

Per @mikhailshilkov, this resource is getting skipped because the REST API does not define a DELETE operation.

[mikhailshilkov]

Indeed, we need to understand the semantics for deleting this resource (it could be a DELETE that is missing or a PUT with a default body) but I don't see it yet

[TarekSalha]

Hi, I am the guy, that asked @clstokes originally about the problem within a support ticket. As the resource currently is defined below ADF in the resource graph, it is typically deleted together with that resource. I opened also a MS ticket about the problem. I will post the results into this thread, then.

[mikhailshilkov]

deleted together with that resource

Our resource model works in the way that when you define the resource, deploy it, then remove the resource, and re-deploy again, you should end up in the same state as at the beginning. Deleting with parent isn't enough for this scenario.

Maybe, we can add exceptions to this rule but I'd love to understand if we can avoid doing that, or if there is a strong principle for when it's okay.

[TarekSalha]

Hi guys, I had a quick call with the MS support today. They will now take the issue to the product group. Let's see, what they will answer. Stay tuned!

[TarekSalha]

quick update on the issue: I am still "fighting" with MS on extracting some useful information. Their point of view is currently: We don’t support delete of managed vnet but there is no need to delete the managed vnet, its the managed PEs that you need to delete. Vnet resource itself does not have to be deleted as it does not emit any charges.

Let's see, maybe, I need to approach the product group not via support requests, but rather via other media... :-|

[TarekSalha]

Might it be the case, that managed vnet also falls under issue 317

[mikhailshilkov]

@TarekSalha if so, there should be some sensible "default" state which we could set for the resource to cancel its "creation". I don't think that sending an empty request body would "delete" it?

[TarekSalha]

This is, what Azure ARM exports me, when configuring managed vnet over the portal. Does that suffice to derive a default state? (I left out the actual params to shorten the code block)

"resources": [
                {
                    "condition": "[and(equals(parameters('version'), 'V2'), parameters('vNetEnabled'))]",
                    "name": "[concat(parameters('name'), '/default')]",
                    "type": "Microsoft.DataFactory/factories/managedVirtualNetworks",
                    "apiVersion": "2018-06-01",
                    "properties": {},
                    "dependsOn": [
                        "[concat('Microsoft.DataFactory/factories/', parameters('name'))]"
                    ]
                },
                {
                    "condition": "[and(equals(parameters('version'), 'V2'), parameters('vNetEnabled'))]",
                    "name": "[concat(parameters('name'), '/AutoResolveIntegrationRuntime')]",
                    "type": "Microsoft.DataFactory/factories/integrationRuntimes",
                    "apiVersion": "2018-06-01",
                    "properties": {
                        "type": "Managed",
                        "managedVirtualNetwork": {
                            "referenceName": "default",
                            "type": "ManagedVirtualNetworkReference"
                        },
                        "typeProperties": {
                            "computeProperties": {
                                "location": "AutoResolve"
                            }
                        }
                    },
                    "dependsOn": [
                        "[concat('Microsoft.DataFactory/factories/', parameters('name'))]",
                        "[concat('Microsoft.DataFactory/factories/', parameters('name'), '/managedVirtualNetworks/default')]"
                    ]
                }
            ]
        },
        {
            "condition": "[equals(parameters('version'), 'V1')]",
            "apiVersion": "2015-01-01-preview",
            "name": "[parameters('name')]",
            "location": "[if(equals(parameters('version'), 'V1'), parameters('location'), 'eastus')]",
            "type": "Microsoft.DataFactory/dataFactories",
            "properties": {},
            "tags": "[ if(contains(parameters('tagsByResource'), 'Microsoft.DataFactory/dataFactories'), parameters('tagsByResource')['Microsoft.DataFactory/dataFactories'], json('{}')) ]"
        }
    ]

[mikhailshilkov]

@TarekSalha Did you shorten the Microsoft.DataFactory/factories/managedVirtualNetworks or was it all empty "properties": {},? Is it before you configured the managed network or after? Are you able to "uncofigure" the managed virtual network - it would be useful to see what that does with the ARM template.

[TarekSalha]

@mikhailshilkov regarding your questions:

Did you shorten the Microsoft.DataFactory/factories/managedVirtualNetworks or was it all empty "properties": {},?

No, this is the exact description of the properties. Weird, right?

Is it before you configured the managed network or after?

This is the resulting ARM template, that I extracted before provisioning the whole ADF (including enabled managed vnet). Thus, the correct answer to your question would be: I did not configure anything.

Here is a "configured example" created a data factory without vnet:

PS C:\Users\tsalha> az datafactory show --resource-group testpulumi --name testpulumi
{
  "additionalProperties": null,
  "createTime": "2021-08-09T18:45:14.464753+00:00",
  "eTag": "\"bf01b3f9-0000-0d00-0000-611177ba0000\"",
  "encryption": null,
  "globalParameters": null,
  "id": "/subscriptions/30d6a2d2-56fa-401b-9a13-9aef8a3264ff/resourceGroups/testpulumi/providers/Microsoft.DataFactory/factories/testpulumi",
  "identity": {
    "principalId": "ceed620c-82b0-45e6-8208-7ab435271d02",
    "tenantId": "a5d051de-762f-4405-9401-3201d0670f2c",
    "type": "SystemAssigned",
    "userAssignedIdentities": null
  },
  "location": "westeurope",
  "name": "testpulumi",
  "provisioningState": "Succeeded",
  "publicNetworkAccess": "Enabled",
  "repoConfiguration": null,
  "resourceGroup": "testpulumi",
  "tags": {},
  "type": "Microsoft.DataFactory/factories",
  "version": "2018-06-01"
}

now, I configure the vnet:

PS C:\Users\tsalha> az rest -m put --header "Content-Type=application/json" -u ‘https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/testpulumi/providers/Microsoft.DataFactory/factories/testpulumi/managedVirtualNetworks/tarek?api-version=2018-06-01' --body "{""properties: {}}"
{
  "etag": "4a01f27c-0000-0d00-0000-61118d320000",
  "id": "/subscriptions/30d6a2d2-56fa-401b-9a13-9aef8a3264ff/resourceGroups/testpulumi/providers/Microsoft.DataFactory/factories/testpulumi/managedvirtualnetworks/tarek",
  "name": "tarek",
  "properties": {
    "alias": "testpulumi",
    "preventDataExfiltration": false,
    "vNetId": "d2d973a8-0155-4a96-ba06-3fa800a7465e"
  },
  "resourceGroup": "testpulumi",
  "type": "Microsoft.DataFactory/factories/managedvirtualnetworks"
}

Be aware, that the properties are all preset with defaults as I did not include any in my request

after reloading the UI I got this:

Load failed
Failed to load resources of type "managedPrivateEndpoint". Some resources may not be shown.
Error: Resource not found. ResourceId: '/subscriptions/30d6a2d2-56fa-401b-9a13-9aef8a3264ff/resourceGroups/testpulumi/providers/Microsoft.DataFactory/factories/testpulumi/managedvirtualnetworks/default'.

Thus API accepts any instance, but UI accepts only default instance.

Are you able to "uncofigure" the managed virtual network - it would be useful to see what that does with the ARM template.

No, I am not able to unconfigure it via the UI or the API.

[mikhailshilkov]

No, I am not able to unconfigure it via the UI or the API.

Yeah, that's the main blocking point for us. So far, we assume that a deletion operation cancels the effect of the creation for any azure-native resource. I'm curious what you'd expect to happen with this (hypothetical) resource on deletion?

[TarekSalha]

I think, currently via UI, you can set this only ones at the beginning at creation time of the ADF and after that, it is read-only ever after. Thus, would it be possible to circumvent the usual mirroring of the azure API's and add a flag "ManagedVnetEnabled" to the ADF resource instead? It would then add the default vnet if enabled and any later changes would require a delete operation of the adf itself. Dioes that sound fair?

[mikhailshilkov]

This is a reasonable line of thinking. However, it goes a bit off the rails of the current provider design where our principle is "project the ARM API directly and respect their resource structure". Let's see what we can do.

[TarekSalha]

just for completeness: As the feature is still public preview in Azure the MSFT prduct group will not be commenting on the topic for now. See their answer: We would like to inform you that, since the ADF managed Vnet documentation is currently under preview state ( Refer doc for same: Managed virtual network & managed private endpoints - Azure Data Factory | Microsoft Docs) and also couldn't get any positive response from PG to comment on the discussion happening, we suspect that due to the same policy restrictions they will not be able to comment there [on github].

[rvvincelli]

Hi all! I've come across this issue on your project while trying out private endpoints on ADF. Indeed, it really seems that:

• they do expect the managed IR vnet to be called default, at least on the UI
• if you call it something else stuff would still work, but the private endpoints won't show up in under the Managed tab
• the vnet resource may not be deleted, even if unreferenced (i.e. no private endpoints rely on that)

Thanks,

[aureq]

@mikhailshilkov Do we know if this is something that has been fixed upstream or are we still awaiting for Azure?

[mikhailshilkov]

@aureq I haven't seen the changes that would fix this for us upstream. If it was, we'd get the resource automatically.

[gio-pcb]

@mikhailshilkov are there any plans of proceeding with this idea?

317

looking to use pulumi to automate some data pipelines but I think we are hitting this same issue:

  azure-native:datafactory:IntegrationRuntime (integrationRuntime):
    error: autorest/azure: Service returned an error. Status=400 Code="ManagedVNetReferencedNotExist" Message="Invalid reference to the managed Virtual Network 'default'. The managed Virtual Network does not exist."