deploys fail due to missing version 2.3.0

[Jackbennett]

Which version of blobfuse was used?

az aks upgrade --enable-blob-driver

Which OS distribution and version are you using?

AKS upgraded to latest nodes default nodes and k8s version 1.28.9

az version
{
  "azure-cli": "2.61.0",
  "azure-cli-core": "2.61.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {
    "k8s-configuration": "2.0.0",
    "k8s-extension": "1.6.1"
  }
}

#  az aks update --enable-blob-driver --name $N --resource-group $RG
Please make sure there is no open-source Blob CSI driver installed before enabling. (y/N): y
...
agentPoolProfiles.nodeImageVersion: AKSUbuntu-2204gen2containerd-202405.27.0

If relevant, please share your mount command.

?

What was the issue encountered?

Warning FailedMount 8s kubelet MountVolume.MountDevice failed for volume "blob-testing" : rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 *** blobfuse2: A new version [2.3.0] is available. Consider upgrading to latest version for bug-fixes & new features. ***

Have you found a mitigation/solution?

No,

Where is any of the supported versions in the published Microsoft AKS nodes documented? I don't see anything. I've just run these commands out-of-the-box. No version upgrades mentioned in https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/storage/mounting-azure-blob-storage-container-fail I don't even know if this is the open source one or not from the install documentation.

[vibhansa-msft]

@andyzhangx can you help on this AKS update related issue please.

[Jackbennett]

If it helps I was on aks 1.26 when I hit the error, I upgraded through 1.27 -> 1.28 and then re-ran az aks upgrade --enable-blob-driver but I always logged this error.

[andyzhangx]

this message blobfuse2: A new version [2.3.0] is a warning message, is not related to the mount error. @Jackbennett could you follow this guide to get more detailed error msg? https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#case2-volume-mountunmount-failed

[Jackbennett]

Ok so that warning threw me off form the next error that was the actual issue, thanks.

Long story shot then I had to add the AKS public IP to the storage accounts firewall allow list, that was failing any authentication regardless of SAS config.

Next it's required to use mountOptions: -o allow_other within the PV NOT the storageClass as documented (that didn't do anything). from the SC is wasn't showing up in the container mount options for the fuse path.