Hardcoded virtualnetworklink suffix causes issues if virtualnetworklink already exists with different name

frank-m commented 2 months ago

What happened:

https://github.com/kubernetes-sigs/cloud-provider-azure/blob/3e881836b693575d3c541cbe6e63186596b162b3/pkg/provider/azure_storageaccount.go#L365-L375

This piece of code assumes that all virtual networks have a name with the '-vnetlink' suffix.

If this is not the case it will try to create the vnetlink, but if a different link already exists for the same vnet. It will fail.

What you expected to happen:

The code should get all virtualnetworklinks and check if one already exists for that vnet (preferably with the Resource ID of the vNet). If it exists, it should not try to create a new virtualnetworklink as only one can exist per vnet.

How to reproduce it (as minimally and precisely as possible):

We hit this issue while upgrading from AKS 1.27 --> 1.28. This also upgrades the CSI drivers, which use this provider. Initially you notice no issues, but when you try to do any actions on your PVC it will report in the logs with errors about not being able to create the virtualnetworklink.

Anything else we need to know?:

The workaround is to manually delete the vnetlink and recreate it with the '-vnetlink' suffix, however that causes a temporary failure in name resolution (depending on TTL of records)

Environment:

Kubernetes version (use kubectl version): happened after upgrading from AKS 1.27 --> 1.28.9
Cloud provider or hardware configuration: Azure AKS
OS (e.g: cat /etc/os-release): Ubuntu
Kernel (e.g. uname -a): -
Install tools: -
Network plugin and version (if this is a network-related bug):-
Others:

Mik4sa commented 1 month ago

I think I have the same issue. My upgrade path was 1.27.3 > 1.28.x (don't know which exact patch version > 1.29.5

This is the error I get on a newly created pvc:

Events:
  Type     Reason                Age               From                                                                                               Message
  ----     ------                ----              ----                                                                                               -------
  Normal   ExternalProvisioning  10s               persistentvolume-controller                                                                        Waiting for a volume to be created either by the external provisioner 'file.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          2s (x4 over 10s)  file.csi.azure.com_csi-azurefile-controller-[...]  External provisioner is provisioning volume for claim "[...]"
  Warning  ProvisioningFailed    1s (x4 over 10s)  file.csi.azure.com_csi-azurefile-controller-[...]  failed to provision volume with StorageClass "my-custom-storage-class": rpc error: code = Internal desc = failed to ensure storage account: get virtual link for vnet(aks-vnet-14154989) and DNS Zone(privatelink.file.core.windows.net) in resourceGroup([...]) returned with GET http://localhost:7788/subscriptions/[...]/resourceGroups/[...]/providers/Microsoft.Network/privateDnsZones/privatelink.file.core.windows.net/virtualNetworkLinks/aks-vnet-14154989-vnetlink
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
  "code": "NotFound",
  "message": "The virtual network link 'aks-vnet-14154989-vnetlink' does not exist for Private DNS zone 'privatelink.file.core.windows.net' in resource group '[...]' of subscription '[...]'."
}
--------------------------------------------------------------------------------

Mik4sa commented 1 month ago

After updating to 1.29.6 I get this error message:

  Warning  ProvisioningFailed  119s                 file.csi.azure.com_csi-azurefile-controller-[...]  failed to provision volume with StorageClass "my-custom-storage-class": rpc error: code = Internal desc = failed to ensure storage account: get private dns zone privatelink.file.core.windows.net returned with GET http://localhost:7788/subscriptions/[...]/resourceGroups/[...]/providers/Microsoft.Network/privateDnsZones/privatelink.file.core.windows.net
--------------------------------------------------------------------------------
RESPONSE 401: 401 Unauthorized
ERROR CODE: ExpiredAuthenticationToken
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ExpiredAuthenticationToken",
    "message": "The access token expiry UTC time '8/2/2024 5:22:59 AM' is earlier than current UTC time '8/2/2024 7:31:24 AM'."
  }
}
--------------------------------------------------------------------------------

Mik4sa commented 1 month ago

After updating to 1.29.7 I was back to the error as in 1.29.5

I recreated the link now with the currently expected name. It worked then as expected.

kubernetes-sigs / cloud-provider-azure