search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.3k stars 21.48k forks source link

Once a private endpoint has been assigned to a subnet; it cannot be removed #39634

Closed tonyt73 closed 1 year ago

tonyt73 commented 5 years ago

https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/private-link/create-private-endpoint-portal.md

When you create a new subnet and assign a private endpoint to it. If you then remove the private endpoint, private link and the association to the vnet subnet and then try to add service endpoint definitions, you will get the following error. Set-AzVirtualNetwork : Private endpoint network policies cannot be enabled on private endpoint subnet Even when the subnet clearly has no private endpoints and the private endpoint and link are deleted. Also trying to change the PrivateEndpointNetworkPolicies flag back to Enabled will also result in the same error.

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

tonyt73 commented 5 years ago

Temporary solution:

msrini-MSFT commented 5 years ago

@tonyt73 , Can you post the output of the below PS command?

$VNET= Get-AzVirtualNetwork -Name "Name of your Vnet" -ResourceGroupName "RG Name"

Get-AzVirtualNetworkSubnetConfig -Name "Name of your Subnet" -VirtualNetwork $VNET

msrini-MSFT commented 5 years ago

@tonyt73 , do you have any update on this issue?

tonyt73 commented 5 years ago

You'll notice that the Internal subnet does not have any private endpoints defined. From either with the virtual network object or the virtual network subnet config object.

Yet, Azure portal and the PS Set-AzVirtualNetwork commands seem to think it does.

Also worth mentioning that you can't change the policy items either 

"PrivateEndpointNetworkPolicies": "Disabled",
"PrivateLinkServiceNetworkPolicies": "Enabled"

Trying to change Disabled to Enabled with also fail.


Name                   : myvnet
ResourceGroupName      : myvnetresourcegroup
Location               : northcentralus
Id                     : /subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet
ResourceGuid           : guid
ProvisioningState      : Succeeded
AddressSpace           : {
                           "AddressPrefixes": [
                             "10.112.0.0/16"
                           ]
                         }
DhcpOptions            : {
                           "DnsServers": [
                             "10.112.2.4",
                             "10.112.2.5"
                           ]
                         }
Subnets                : [
                           {
                             "Delegations": [],
                             "Name": "GatewaySubnet",
                             "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/GatewaySubnet",
                             "AddressPrefix": [
                               "10.112.100.224/27"
                             ],
                             "IpConfigurations": [
                               {
                                 "Id": "/subscriptions/{subid}/resourceGroups/myvnet/providers/Microsoft.Network/virtualNetworkGateways/gateway/ipConfigurations/default"
                               }
                             ],
                             "ServiceAssociationLinks": [],
                             "ResourceNavigationLinks": [],
                             "ServiceEndpoints": [],
                             "ServiceEndpointPolicies": [],
                             "PrivateEndpoints": [],
                             "ProvisioningState": "Succeeded",
                             "PrivateEndpointNetworkPolicies": "Enabled",
                             "PrivateLinkServiceNetworkPolicies": "Enabled"
                           },
                           {
                             "Delegations": [],
                             "Name": "ServiceHosts",
                             "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/ServiceHosts",
                             "AddressPrefix": [
                               "10.112.100.0/29"
                             ],
                             "IpConfigurations": [
                               {
                                 "Id": "/subscriptions/{subid}/resourceGroups/myvm/providers/Microsoft.Network/networkInterfaces/myvmnic/ipConfigurations/ipconfig1"
                               }
                             ],
                             "ServiceAssociationLinks": [],
                             "ResourceNavigationLinks": [],
                             "NetworkSecurityGroup": {
                               "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/networkSecurityGroups/mynsg"
                             },
                             "ServiceEndpoints": [],
                             "ServiceEndpointPolicies": [],
                             "PrivateEndpoints": [],
                             "ProvisioningState": "Succeeded",
                             "PrivateEndpointNetworkPolicies": "Enabled",
                             "PrivateLinkServiceNetworkPolicies": "Enabled"
                           },
                           {
                             "Delegations": [],
                             "Name": "External",
                             "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/External",
                             "AddressPrefix": [
                               "10.112.1.0/24"
                             ],
                             "IpConfigurations": [
                               {
                                 "Id": "/subscriptions/{subid}/resourceGroups/mywebvm/providers/Microsoft.Network/networkInterfaces/mywebvmnic/ipConfigurations/ipconfig1"
                               },
                               {
                                 "Id": "/subscriptions/{subid}/resourceGroups/myvm2/providers/Microsoft.Network/networkInterfaces/myvm2nic/ipConfigurations/ipconfig1"
                               },
                             ],
                             "ServiceAssociationLinks": [],
                             "ResourceNavigationLinks": [],
                             "NetworkSecurityGroup": {
                               "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/networkSecurityGroups/NCU_NSG_External"
                             },
                             "ServiceEndpoints": [
                               {
                                 "ProvisioningState": "Succeeded",
                                 "Service": "Microsoft.AzureCosmosDB",
                                 "Locations": [
                                   "*"
                                 ]
                               },
                               {
                                 "ProvisioningState": "Succeeded",
                                 "Service": "Microsoft.Sql",
                                 "Locations": [
                                   "northcentralus"
                                 ]
                               },
                               {
                                 "ProvisioningState": "Succeeded",
                                 "Service": "Microsoft.ServiceBus",
                                 "Locations": [
                                   "*"
                                 ]
                               }
                             ],
                             "ServiceEndpointPolicies": [],
                             "PrivateEndpoints": [],
                             "ProvisioningState": "Succeeded",
                             "PrivateEndpointNetworkPolicies": "Disabled",
                             "PrivateLinkServiceNetworkPolicies": "Disabled"
                           },
                           {
                             "Delegations": [],
                             "Name": "Internal",
                             "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/Internal",
                             "AddressPrefix": [
                               "10.112.2.0/24"
                             ],
                             "IpConfigurations": [
                               {
                                 "Id": "/subscriptions/{subid}/resourceGroups/NCU_RG_DNS/p roviders/Microsoft.Network/networkInterfaces/dnsvm/ipConfigurations/ipconfig1"
                               },
                             ],
                             "ServiceAssociationLinks": [],
                             "ResourceNavigationLinks": [],
                             "NetworkSecurityGroup": {
                               "Id": "/subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/networkSecurityGroups/NCU_NSG_Internal"
                             },
                             "ServiceEndpoints": [],
                             "ServiceEndpointPolicies": [],
                             "PrivateEndpoints": [],
                             "ProvisioningState": "Succeeded",
                             "PrivateEndpointNetworkPolicies": "Disabled",
                             "PrivateLinkServiceNetworkPolicies": "Disabled"
                           }
                         ]
EnableDdosProtection   : false
DdosProtectionPlan     : null

> Get-AzVirtualNetworkSubnetConfig -name Internal -VirtualNetwork $vn

Name                              : Internal
Id                                : /subscriptions/{subid}/resourceGroups/myvnetresourcegroup/providers/Microsoft.Network/virtualNetworks/myvnet/subnets/Internal
ProvisioningState                 : Succeeded
AddressPrefix                     : {10.112.2.0/24}
IpConfigurations                  : [
                                      {
                                        "Id": "/subscriptions/{subid}/resourceGroups/NCU_RG_DNS/providers/Microsoft.Network/networkInterfaces/dnsvm/ipConfigurations/ipconfig1"
                                      }
                                    ]
ResourceNavigationLinks           : []
ServiceAssociationLinks           : []
NetworkSecurityGroup              : {
                                      "Id": "/subscriptions/{subid}/resourceGroups/NCU_RG_Network/providers/Microsoft.Network/networkSecurityGroups/NCU_NSG_Internal"
                                    }
RouteTable                        : null
NatGateway                        : null
ServiceEndpoints                  : []
ServiceEndpointPolicies           : []
PrivateEndpoints                  : []
PrivateEndpointNetworkPolicies    : Disabled
PrivateLinkServiceNetworkPolicies : Disabled
msrini-MSFT commented 5 years ago

@KumudD , Can you take a look at this issue and update the doc with the alternative ?

asudbring commented 4 years ago

assign:@malopMSFT

malopMSFT commented 4 years ago

Hi tonyt73

We had this into our bug fixes list, do you mind trying to replicate the issue and see if it's still active It should be solved by now with one of our previous releases

rtaylor72 commented 4 years ago

I just got this error as part of my build process. I can no longer run "Set-AzVirtualNetwork" on a loaded vnet config (Get-AzVirtualNetwork) if that vnet has a subnet with a private endpoint attached to it. I have also narrowed it down to when I run "Set-AzVirtualNetworkSubnetConfig" with the "-PrivateLinkServiceNetworkPoliciesFlag 'Enabled'" setting causes the error.

rtaylor72 commented 4 years ago

Circling back around to this... a little confusing, but I came across something stating PrivateLinkServiceNetworkPoliciesFlag has to be 'Disabled' in order to add a private end point... seems not intuitive.

CloudPlumber commented 4 years ago

Hey everybody,

I ran into the same issue, when I deployed a private endpoint into the subnet via pipeline and set this setting in a template. After deleting it in the template the error stayed and now I always get the error when I execute the pipeline:

"PrivateEndpointNetworkPoliciesCannotBeEnabledOnPrivateEndpointSubnet\",\"message\":\"Private endpoint network policies cannot be enabled on private endpoint subnet /subscriptions/%subscriptionID%/resourceGroups/%RG%/providers/Microsoft.Network/virtualNetworks/%VNETNAME%/subnets/%SUBNETNAME%.

kevinwedwards commented 4 years ago

We're also seeing this issue with our vnet arm template config, after deploying private link for sql server via another arm template. Feels similar to the error above.

malopMSFT commented 4 years ago

Hi guys,

Appreciate if you can raise a support ticket, it's hard t diagnose multiple issues without looking at the configuration.

I'm reading multiple issues listed here with PrivateEndpointNetworkPoliciesCannotBeEnabledOnPrivateEndpointSubnet and PrivateLinkServiceNetworkPoliciesFlag as 2 separate independent controls.

markmcdon7 commented 3 years ago

Did anyone ever raise a support ticket and get a fix from them?

mdehaas commented 3 years ago

We did but it did not lead to a satisfying result. Somewhere along the process also we decided this scenario does not fit into our Modern Workplace offering. We are only using SMB3 shares in Azure as part of an Application Resource Group for storing the application data (as was once the intended purpose from Microsoft). The only landing zones for data which has to be accessed from end-user devices are OneDrive and Sharepoint Online. We have not seen a case from our customers which forced us into another route (Customers just need some convincing that shifting a share to Azure is not 'modern')

gowrisankar22 commented 3 years ago

@msrini-MSFT @SaiKishor-MSFT I am also facing the same issue. I Unable to change the PrivateEndpointNetworkPolicies

For SubnetProperties I want to disable the value forPrivateEndpointNetworkPolicies and PrivateLinkServiceNetworkPolicies but the below sample code is not disabling that. Can you help ?


const (
    PrivateEndpointNetworkPolicies string = "Disabled"
)

var penp = PrivateEndpointNetworkPolicies

SubnetPropertiesFormat: &network.SubnetPropertiesFormat{
       AddressPrefix:                  to.StringPtr(cidr),
    PrivateEndpointNetworkPolicies: &penp,
    PrivateLinkServiceNetworkPolicies: &penp,
}
asudbring commented 3 years ago

Sending to PM for review

assign:@ivapplyr

reborn1867 commented 3 years ago

Hi @malopMSFT, Any update on this? I'm facing the same issue and I cannot understand why this issue kept being ignored? It's pretty straight forward and there must be a better approach to handle it.

malopMSFT commented 3 years ago

Please raise a support ticket for proper troubleshooting and diagnosis of the issue, is not clear on this open item what is the issue with out documentation that describes those properties and behavior.

For active issues on deployments, support ticket is the right method as many times requires specific investigation onto a particular customer deployment.

reborn1867 commented 3 years ago

@malopMSFT Honestly, I don't think this issue has anything to do with "particular customer deployment", the symptom is easy to reproduce:

Do you think this is as expected?

malopMSFT commented 3 years ago

Is expected, Users must get an error when deploying private endpoints and the property is enabled.

The sequence can be any of the following: 1) subnet with PEPolicies=Enables then deploy a PE => Error 2) Subnet has PE (because PEPolicies are disabled) then updating the subnet to PEPolicies=Enabled => Error

The property is mainly designed to provide a toggle once NSG is supported, currently in public preview.

For preview subscriptions, this error does not show and instead, NSG will start to apply to the private endpoints

Details can be found here: https://docs.microsoft.com/en-us/azure/private-link/disable-private-endpoint-network-policy

apeschel commented 2 years ago

Confirming this bug is still active as of Dec 21, 2021.

prithvy commented 2 years ago

Confirming this bug is still active as of JAN 7, 2022.

ivapplyr commented 2 years ago

Hello Folks,

For those who are receiving this error message, it is per design. Please read documentation below to have access to the Public Preview of Private Link NSG & UDR :)

https://azure.microsoft.com/en-us/updates/private-link-nsg-support-extended-regional-availability/ https://azure.microsoft.com/en-us/updates/extended-regional-availability-for-private-link-udr-support/

ivapplyr commented 2 years ago

@tonyt73 would we be able to close this as there are multi thread of issues prior to 2021

IPvPho commented 2 years ago

When trying to disable the Private Link Service POlicies via Az CLI, we are now getting this error: "however, it does not have permission to perform action 'Microsoft.Network/routeTables/join/action' on the linked scope(s)"

We understand the related error about the policies being enabled when trying to add a PLS, but we are not sure why this is trying to read/write to the route table.

asudbring commented 2 years ago

@IPvPho Can you provide more information on the steps and environment when getting this error? Is this a new deployment of private endpoint to an existing subnet with the policies applied? Are there any permissions or policies deployed in your environment that would prevent your account from updating route tables? Can you provide the full error you are receiving and possibly a screenshot?

Thanks!

@ivapplyr do you know why it would error out on updating the route table?

asudbring commented 1 year ago

Thank you for you dedication to our documentation.

Unfortunately, we have been unable to review this issue in a timely manner. We sincerely apologize for the delayed response. We are closing this issue. If you feel that the problem persists, please respond to this issue with additional information.

Please continue to provide feedback about the documentation. We appreciate your contributions to our community.

please-close