search

Azure / azure-storage-fuse

A virtual file system adapter for Azure Blob storage
Other
659 stars 206 forks source link

Authentication failure with MSI in creating persistent volume with azure blob storage in AKS #1523

Closed himanshigarg closed 3 weeks ago

himanshigarg commented 1 month ago

Which version of blobfuse was used?

blobfuse2 version 2.3.0

Which OS distribution and version are you using?

OS-IMAGE : CBL-Mariner/Linux KERNEL-VERSION: 5.15.164.1-1.cm2

If relevant, please share your mount command.

Following this (https://learn.microsoft.com/en-us/azure/aks/azure-csi-blob-storage-provision?tabs=mount-nfs%2Csecret) document to mount azure blob storage as persistent volume in aks

Below is the config for persistent volume and persistent volume claim: apiVersion: v1 kind: PersistentVolume metadata: name: {{ .Release.Namespace }}-airflow2dagpv labels: usage: {{ .Release.Namespace }}-airflowtwo-dag spec: capacity: storage: 5Gi accessModes:

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: airflow2dagpvc spec: accessModes:

What was the issue encountered?

I am not able to mount azure blob storage as persistent volume in AKS

Have you found a mitigation/solution?

No

Please share logs if available.

E0919 05:04:33.435872 219263 nodeserver.go:412] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 05:04:33.435980 219263 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 05:04:32.692058 266161 nodeserver.go:412] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 05:04:32.692184 266161 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 05:04:32.554146 198544 nodeserver.go:412] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 05:04:32.554266 198544 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. Streaming logs below: I0918 13:21:14.333490 508225 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:14.333701 508225 utils.go:105] GRPC request: {} I0918 13:21:14.356419 508225 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:14.650328 508225 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:14.650603 508225 utils.go:105] GRPC request: {} I0918 13:21:14.650882 508225 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:14.802771 508225 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:14.802813 508225 utils.go:105] GRPC request: {} I0918 13:21:14.802943 508225 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss000003"} Streaming logs below: I0918 13:21:12.460098 511207 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:12.460130 511207 utils.go:105] GRPC request: {} I0918 13:21:12.463979 511207 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:12.840527 511207 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:12.840555 511207 utils.go:105] GRPC request: {} I0918 13:21:12.840637 511207 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:12.951682 511207 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:12.951711 511207 utils.go:105] GRPC request: {} I0918 13:21:12.951825 511207 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss000007"} Streaming logs below: I0918 13:21:06.819270 423909 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:06.819302 423909 utils.go:105] GRPC request: {} I0918 13:21:06.829853 423909 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:06.883370 423909 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:06.883395 423909 utils.go:105] GRPC request: {} I0918 13:21:06.883450 423909 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:07.230935 423909 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:07.230957 423909 utils.go:105] GRPC request: {} I0918 13:21:07.231058 423909 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss00000e"} E0919 05:04:32.618429 202095 nodeserver.go:412] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 05:04:32.618537 202095 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 05:00:26.345010 508869 nodeserver.go:412] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 05:00:26.345128 508869 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. Streaming logs below: I0918 13:21:10.631138 511685 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:10.631169 511685 utils.go:105] GRPC request: {} I0918 13:21:10.633441 511685 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:10.805804 511685 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:10.805866 511685 utils.go:105] GRPC request: {} I0918 13:21:10.805964 511685 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:11.057546 511685 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:11.057577 511685 utils.go:105] GRPC request: {} I0918 13:21:11.057705 511685 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss000008"} E0919 05:04:32.829736 247282 nodeserver.go:412] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 05:04:32.829844 247282 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 04:34:00.391918 510570 nodeserver.go:412] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 04:34:00.392068 510570 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. Streaming logs below: I0918 13:21:14.514378 506943 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:14.514427 506943 utils.go:105] GRPC request: {} I0918 13:21:14.517871 506943 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:14.700040 506943 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:14.700067 506943 utils.go:105] GRPC request: {} I0918 13:21:14.700227 506943 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:15.681724 506943 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:15.681749 506943 utils.go:105] GRPC request: {} I0918 13:21:15.681900 506943 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss000000"} Streaming logs below: I0918 13:21:12.417317 507029 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:12.417350 507029 utils.go:105] GRPC request: {} I0918 13:21:12.419020 507029 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:12.656412 507029 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:12.656448 507029 utils.go:105] GRPC request: {} I0918 13:21:12.656585 507029 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:13.215683 507029 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:13.215829 507029 utils.go:105] GRPC request: {} I0918 13:21:13.216027 507029 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss000002"} Streaming logs below: I0918 13:21:06.171598 428161 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:06.172094 428161 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:06.171626 428161 utils.go:105] GRPC request: {} I0918 13:21:06.176528 428161 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:06.172131 428161 utils.go:105] GRPC request: {} I0918 13:21:06.176930 428161 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:06.847992 428161 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:06.848020 428161 utils.go:105] GRPC request: {} I0918 13:21:06.848132 428161 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss00000c"} E0919 05:04:33.826022 203631 nodeserver.go:412] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. E0919 05:04:33.826138 203631 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 blobfuse2: A new version [2.3.2] is available. Consider upgrading to latest version for bug-fixes & new features. Visit https://aka.ms/blobfuse2warnings#230 to see the list of vulnerabilities associated with your current version [2.3.0] Error: failed to initialize new pipeline [failed to authenticate credentials for azstorage] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. Streaming logs below: I0918 13:21:14.178942 506312 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:14.178985 506312 utils.go:105] GRPC request: {} I0918 13:21:14.181604 506312 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:14.394539 506312 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:14.394568 506312 utils.go:105] GRPC request: {} I0918 13:21:14.394639 506312 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:14.969844 506312 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:14.969865 506312 utils.go:105] GRPC request: {} I0918 13:21:14.969974 506312 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss000004"} Streaming logs below: I0918 13:21:05.311569 337952 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:05.311611 337952 utils.go:105] GRPC request: {} I0918 13:21:05.314295 337952 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:05.490317 337952 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:05.490384 337952 utils.go:105] GRPC request: {} I0918 13:21:05.490481 337952 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:06.092211 337952 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:06.092243 337952 utils.go:105] GRPC request: {} I0918 13:21:06.092393 337952 utils.go:111] GRPC response: {"node_id":"aks-azespool0-25990732-vmss000000"} Streaming logs below: I0918 13:21:11.122802 510279 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:11.122856 510279 utils.go:105] GRPC request: {} I0918 13:21:11.129196 510279 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:11.303565 510279 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:11.303595 510279 utils.go:105] GRPC request: {} I0918 13:21:11.303677 510279 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:12.010403 510279 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:12.010450 510279 utils.go:105] GRPC request: {} I0918 13:21:12.010618 510279 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss000001"} Streaming logs below: I0918 13:21:06.611438 278682 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:06.611467 278682 utils.go:105] GRPC request: {} I0918 13:21:06.612722 278682 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:06.686497 278682 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:06.686532 278682 utils.go:105] GRPC request: {} I0918 13:21:06.686820 278682 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:07.067038 278682 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:07.067066 278682 utils.go:105] GRPC request: {} I0918 13:21:07.067159 278682 utils.go:111] GRPC response: {"node_id":"aks-azespool0-25990732-vmss000001"} Streaming logs below: I0918 13:21:06.460125 467235 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:06.460163 467235 utils.go:105] GRPC request: {} I0918 13:21:06.462358 467235 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:06.466027 467235 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:06.466052 467235 utils.go:105] GRPC request: {} I0918 13:21:06.466105 467235 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:06.663481 467235 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:06.663510 467235 utils.go:105] GRPC request: {} I0918 13:21:06.663622 467235 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss00000d"} Streaming logs below: I0918 13:21:13.978940 506163 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:13.978984 506163 utils.go:105] GRPC request: {} I0918 13:21:13.982052 506163 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:14.146254 506163 utils.go:104] GRPC call: /csi.v1.Identity/GetPluginInfo I0918 13:21:14.146282 506163 utils.go:105] GRPC request: {} I0918 13:21:14.146355 506163 utils.go:111] GRPC response: {"name":"blob.csi.azure.com","vendor_version":"v1.23.7"} I0918 13:21:15.114753 506163 utils.go:104] GRPC call: /csi.v1.Node/NodeGetInfo I0918 13:21:15.114780 506163 utils.go:105] GRPC request: {} I0918 13:21:15.114901 506163 utils.go:111] GRPC response: {"node_id":"aks-azinternal-21860637-vmss000006"}

souravgupta-msft commented 1 month ago

Hi @himanshigarg. Thanks for reaching out. Please upgrade to version 2.3.2 which contains critical bug fixes. Also please share the Blobfuse2 debug level logs.

himanshigarg commented 1 month ago

Hi @souravgupta-msft, Can you tell me how to upgrade to 2.3.2 version. I ran the below command to install csi blob driver in the aks cluster? az aks update --enable-blob-driver --name myAKSCluster --resource-group myResourceGroup

Also for the logs, I have set the log level to DEBUG (- --log-level=LOG_DEBUG in the mountOptions) but still don't see debug logs in the csi-blob-node logs. I ran the below command in the aks cluster to get the logs. kubectl logs -l app=csi-blob-node -n kube-system -c blob

vibhansa-msft commented 1 month ago

@andyzhangx is the latest version of blobfuse (2.3.2) integrated with AKS release?

andyzhangx commented 1 month ago

Hi @souravgupta-msft, Can you tell me how to upgrade to 2.3.2 version. I ran the below command to install csi blob driver in the aks cluster? az aks update --enable-blob-driver --name myAKSCluster --resource-group myResourceGroup

Also for the logs, I have set the log level to DEBUG (- --log-level=LOG_DEBUG in the mountOptions) but still don't see debug logs in the csi-blob-node logs. I ran the below command in the aks cluster to get the logs. kubectl logs -l app=csi-blob-node -n kube-system -c blob

@himanshigarg you could run this command to install v2.3.2 on aks node as a workaround. AKS Sep release (not rolled out yet) would install 2.3.2 by default on the agent node.

kubectl patch daemonset csi-blob-node -n kube-system -p '{"spec":{"template":{"spec":{"initContainers":[{"env":[{"name":"INSTALL_BLOBFUSE2","value":"true"},{"name":"BLOBFUSE2_VERSION","value":"2.3.2"}],"name":"install-blobfuse-proxy"}]}}}}'
andyzhangx commented 1 month ago

btw, the mount failure is not related to blobfuse version. in the pv config, secret is not needed for msi auth type, you could refer to: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/deploy/example/pv-blobfuse-auth.yaml

if there is still issue, pls provide the full csi driver logs, I see you have provided part of csi driver logs which does not contain useful info for troubleshooting.

  AzureStorageAuthType: MSI
  AzureStorageIdentityResourceID:
nodeStageSecretRef:
  name: azure-blob-secret
  namespace: {{ .Release.Namespace }}
himanshigarg commented 1 month ago

@andyzhangx, I am still getting same error even after removing secret from the config. Regarding logs, Can you please guide me how to get the entire logs. I am running below command to get the logs. I shared all the logs returned by this command. kubectl logs -l app=csi-blob-node -n kube-system -c blob

Sharing the new config file: apiVersion: v1 kind: PersistentVolume metadata: name: {{ .Release.Namespace }}-airflow2dagpv labels: usage: {{ .Release.Namespace }}-airflowtwo-dag spec: capacity: storage: 5Gi accessModes:

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: airflow2dagpvc spec: accessModes:

andyzhangx commented 1 month ago

@himanshigarg find guide here about how to get csi driver logs: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/csi-debug.md#case2-volume-mountunmount-failed

himanshigarg commented 1 month ago

@andyzhangx, sharing the logs with csi driver logs. Ran the below command to get the logs. kubectl logs daemonset/csi-blob-node -c blob -n kube-system logs.txt

andyzhangx commented 1 month ago

@himanshigarg the mount error is failed to initialize new pipeline [failed to authenticate credentials for azstorage], have you granted storage account Contributor role to the managed identity osdu-identity-cal3whpwnkdpk?

andyzhangx commented 1 month ago

and make sure osdu-identity-cal3whpwnkdpk is already bound to the agent nodes, you could also use the kubelet identity which is already bound to the node pool, you only need to granted storage account Contributor role to that managed identity

himanshigarg commented 1 month ago

@andyzhangx, I tried using kubelet identity also and assigned Contributor role to this identity over the storage account but still getting same error. Logs are also not very helpful in getting to the root cause. Is there any way I can debug and resolve this issue?

Btw, if you are MSFT employee, can we connect over teams chat to resolve this issue faster?

andyzhangx commented 1 month ago

@himanshigarg pls also get /var/log/blobfuse2.log on the node for troubleshooting, you could find my MS alias in the github profile.

andyzhangx commented 1 month ago

@vikama-microsoft is AzureStorageIdentityResourceID format correct? AzureStorageIdentityResourceID":"/subscriptions/82adc5c1-83bb-47d2-9e9b-da34d6faa530/resourceGroups/Compute-rg-smahg21-dgsqwc/providers/Microsoft.ManagedIdentity/userAssignedIdentities/osdu-identity-cal3whpwnkdpk

I think you could ssh to the node, and try with blobfuse2 mount directly. not sure about whether AzureStorageIdentityResourceID format is correct, you could try AzureStorageIdentityClientID which is more commonly used.

vibhansa-msft commented 4 weeks ago

Is the issue resolved or AKS team is still investigating this ?

himanshigarg commented 4 weeks ago

@vibhansa-msft, I am still facing this issue even after using AzureStorageIdentityClientID. I still need help in debugging this issue.

souravgupta-msft commented 3 weeks ago

Closing as this is resolved. The issue was that the container was not present in the storage account.