Directory creation Inconsistencies with blobfuse2 mount using key authentication

[Kaushik-Vaibhav]

What happened: When mounting an Azure Blob Storage container with blobfuse2 using access-key, directory listing and creation behave inconsistently. Existing directories in the blob container are not visible in the mounted directory, and attempts to create new directories in the mount produce errors, although files can be created successfully.

What you expected to happen: Should be able to create new directories on the mounted blob storage and existing directories on the blob should be visible as well in the mounted directory

How to reproduce it: Mount the blob_storage

mkdir test
export AZURE_STORAGE_ACCOUNT=STORAGE_ACCOUNT
export AZURE_STORAGE_ACCESS_KEY=ACCESS_KEY
blobfuse2 test --container-name=CONTAINER_NAME --tmp-path=/tmp/blobfuse -o allow_other --file-cache-timeout-in-seconds=0 -o direct_io

check the mount and cd to the mounted directory

delphix@vk-hs:~$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
blobfuse2                           28P     0   28P   0% /export/home/delphix/test

But now if I do 'ls' from this mounted dir I don't see the directories created in the blob (they do exist I checked on the portal) delphix@vk-hs:~/test$ ls

and if I try to create a dir, I get an error

delphix@vk-hs:~/test$ sudo mkdir test_dir
mkdir: cannot create directory ‘test_dir’: No such file or directory

But upon checking the azure portal, the said dir has been created. Additionally, I attempted to create a csv file in the mounted dir, which was successful and the file was visible via 'ls' and on the portal.

delphix@vk-hs:~/test$ vi new_file.csv
delphix@vk-hs:~/test$ ls
new_file.csv

Anything else we need to know?:

Environment: CSI Driver version: v1.24.3 Kubernetes version (use kubectl version): v1.27.4 OS (e.g. from /etc/os-release): "Ubuntu", (20.04.6 LTS) (Focal Fossa)" Kernel (e.g. uname -a): Linux vk-azure 5.15.0-1073-dx2024092516-650d167a5-azure https://github.com/kubernetes-sigs/blob-csi-driver/pull/82 SMP Wed Sep 25 16:38:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Install tools: snap (for microk8 installation)

[andyzhangx]

@Kaushik-Vaibhav are you using a Azure DataLake storage account ? if yes, you need to specify the mount option --use-adls=true in the persistent volume.

[Kaushik-Vaibhav]

This seems to work Thanks I'm also trying to setup a kubernetes cluster, mounting the azure_blob to the containers. The pv I'm using to setup the kubernetes cluster is

apiVersion: v1
kind: PersistentVolume
metadata:
  name: stage-storage-pv
  labels:
    io.hyperscale.service: stage-storage-path
spec:
  capacity:
    storage: 10Gi  
  accessModes:
    - ReadWriteMany  
  mountOptions:
    - -o allow_other
    - -o direct_io
    - --file-cache-timeout-in-seconds=0
    - --use-attr-cache=false
    - --invalidate-on-sync=true
    - --attr-timeout=0
    - --entry-timeout=0
    - --negative-timeout=0
    - --use-adls=true

  storageClassName: ""
  csi:
    driver: blob.csi.azure.com
    volumeHandle: azure-csi-driver-volume
    volumeAttributes:
      containerName: CONTAINER_NAME  # Azure Blob Storage container name (same as provisioned VM)
      storageAccount: STORAGE_ACCOUNT # Azure storage account name
      AzureStorageAuthType: key
      AzureStorageAccessKey:
      protocol: fuse2
      #nodeStageSecretRef:
      #  name: azure-secret
      #  namespace: default

If I don't want to create a secret, what would be the volumeAttribute to provide the accessKey? I tried 'AzureStorageAccessKey' in the pv but getting an error in connecting to the blob container

The node driver logs:

I1107 06:38:53.095494   21856 utils.go:104] GRPC call: /csi.v1.Node/NodeStageVolume
I1107 06:38:53.095511   21856 utils.go:105] GRPC request: {"staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/4e9ad32f90ff822edaf1ee997f3d1da95fdf44cd5a5760a3ef571a91fe66a39b/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other","-o direct_io","--file-cache-timeout-in-seconds=0","--use-attr-cache=false","--invalidate-on-sync=true","--attr-timeout=0","--entry-timeout=0","--negative-timeout=0","--use-adls=true"],"volume_mount_group":"50"}},"access_mode":{"mode":5}},"volume_context":{"AzureStorageAccessKey":"ACCESS_KEY","AzureStorageAuthType":"key","containerName":"CONTAINER_NAME","protocol":"fuse2","storageAccount":"STORAGE_ACCOUNT"},"volume_id":"azure-csi-driver-volume"}
I1107 06:38:53.095987   21856 blob.go:457] parsing volumeID(azure-csi-driver-volume) return with error: error parsing volume id: "azure-csi-driver-volume", should at least contain two #
I1107 06:38:53.096012   21856 blob.go:538] volumeID(azure-csi-driver-volume) authEnv: [AZURE_STORAGE_AUTH_TYPE=key]
I1107 06:38:53.099665   21856 blob.go:608] get account(hm3828) key from secret(default, azure-storage-account-hm3828-secret) failed with error: could not get secret(azure-storage-account-STORAGE_ACCOOUNT-secret): secrets "azure-storage-account-STORAGE_ACCOUNT-secret" not found, use cluster identity to get account key instead
I1107 06:38:53.099717   21856 azure_metrics.go:118] "Observed Request Latency" latency_seconds=0.0039818 request="blob_csi_driver_node_stage_volume" resource_group="" subscription_id="" source="blob.csi.azure.com" volumeid="azure-csi-driver-volume" result_code="failed_csi_driver_node_stage_volume"
E1107 06:38:53.099731   21856 utils.go:109] GRPC error: rpc error: code = Internal desc = no key for storage account(hm3828) under resource group(), err StorageAccountClient is nil

[andyzhangx]

the account key can only be stored in secret, so just use nodeStageSecretRef