Mount problem on 1.23.4

[taeviden]

What happened: Blob Container is not mounted in container (files not visible in folder, read-only), but visible as mounted on AKS level.

What you expected to happen: Files from blob container visible and possible to be edited

How to reproduce it: Terraform code;

resource "kubernetes_deployment" "grafana" {

  metadata {
    name      = "grafana"
    namespace = xxx
    labels = {
      app                           = "grafana"
      "azure.workload.identity/use" = "true"
    }
    annotations = {
    }
  }

  spec {
    replicas = 1
    selector {
      match_labels = {
        app = "grafana"
      }
    }

    template {
      metadata {
        labels = {
          app                           = "grafana"
          role                          = "monitoring"
          "azure.workload.identity/use" = "true"
        }
      }

      spec {
        node_selector = {
          "kubernetes.io/os" = "linux"
          "agentpool"        = "xxx"
        }

        service_account_name = xxx

        affinity {
          pod_anti_affinity {
            preferred_during_scheduling_ignored_during_execution {
              weight = 100
              pod_affinity_term {
                label_selector {
                  match_expressions {
                    key      = "app"
                    operator = "In"
                    values   = ["grafana"]
                  }
                }
                topology_key = "kubernetes.io/hostname"
              }
            }
          }
        }
        volume {
          name = "grafana-config-volume"
          config_map {
            name         = "grafana-conf"
            default_mode = "0420"
          }
        }
        volume {
          name = "blob-storage"
          csi {
            driver = "blob.csi.azure.com"
            volume_attributes = {
              storageAccountName = xxx
              secretName = xxx
              secretNamespace = xxx
              containerName  = xxx
              clientID       = xxx
              resourceGroup  = xxx
              mountOptions   = "-o gid=1000"
            }
          }
        }
        security_context {
          fs_group        = 1000
          run_as_non_root = true
          run_as_user     = 1000
        }
        container {
          image             = xxx
          name              = "grafana"
          image_pull_policy = "Always"
          port {
            container_port = xxx
          }
          env {
            name  = "APP_FOLDER"
            value = xxx
          }
          env {
            name  = "GF_PATHS_DATA"
            value = "/usr/src/app/grafana_data"
          }
          env {
            name  = "GF_DATABASE_URL"
            value = "sqlite3:///usr/src/app/grafana_data/grafana.db?cache=private&mode=rwc&_journal_mode=WAL"
          }
          env {
            name  = "GF_PATHS_CONFIG"
            value = "/usr/src/app/conf/grafana.ini"
          }
          volume_mount {
            name       = "grafana-config-volume"
            mount_path = "/usr/src/app/conf"
          }
          volume_mount {
            name              = "blob-storage"
            mount_path        = "/usr/src/app/grafana_data"
            read_only         = false
          }
          resources {
            limits = {
              cpu    = "2"
              memory = "2Gi"
            }
            requests = {
              cpu    = "500m"
              memory = "1Gi"
            }
          }
        }
      }
    }
  }
}

secret:

resource "kubernetes_secret" "grafana_storage" {
  type = "Opaque"

  metadata {
    name      = xxx
    namespace = xxx
  }

  data = {
    azurestorageaccountname = xxx
    azurestorageaccountkey  = xxx
  }

  wait_for_service_account_token = false
}

Anything else we need to know?: Tested with different mountOptions like -o allow_other, or parameters like: mountPermissions = "0777"

inside container: drwxr-xr-x 2 root root 4096 Jul 15 13:05 grafana_data

grafana-f7dc5b56c-t6qf6:~/grafana_data$ ls -la total 8 drwxr-xr-x 2 root root 4096 Jul 15 13:05 . drwx------ 1 app app 4096 Jul 15 13:05 ..

Provided configuration working fine on: AKS:1.27.7 CSI Driver: 1.22.6

inside container: drwxrwxrwx 2 root root 4096 Jul 15 15:39 grafana_data

grafana-65bdf48894-5hpvj:~/grafana_data$ ls -la total 0 drwxrwxrwx 2 root app 4096 Jun 26 16:46 alerting drwxrwxrwx 2 root app 4096 Jun 26 16:46 csv -rwxrwxrwx 1 root app 929792 Jul 15 15:35 grafana.db -rwxrwxrwx 1 root app 32768 Jul 15 15:36 grafana.db-shm -rwxrwxrwx 1 root app 4136512 Jul 15 15:36 grafana.db-wal drwxrwxrwx 2 root app 4096 Jun 26 16:46 png drwxrwxrwx 2 root app 4096 Jun 26 16:46 storage

Both clusters using exactly same code, image, configuration.

Environment:

• CSI Driver version: 1.23.4
• Kubernetes version (use kubectl version): v1.28.9
• OS (e.g. from /etc/os-release): Alpine Linux v3.18
• Kernel (e.g. uname -a):
• Install tools:
• Others: terraform 1.5.1, azurerm 3.99.0, kubernetes provider 2.23.0

[andyzhangx]

Blob Container is not mounted in container (files not visible in folder, read-only), but visible as mounted on AKS level.

@taeviden what does this mean? it works well on AKS? or only works well in one AKS cluster? and could you check whether you are using Azure DataLake Gen2 storage account? details: https://learn.microsoft.com/en-us/azure/aks/azure-csi-blob-storage-provision?tabs=mount-nfs%2Csecret#before-you-begin

[taeviden]

no DataLake storage. When you check pod describe, you can see that storage was mounted, same in AKS events, etc. But when you go inside container then no data.

[andyzhangx]

@taeviden what's the output of kubectl get CSIDriver blob.csi.azure.com -o yaml ? per this doc, it should have tokenRequests field

https://github.com/kubernetes-sigs/blob-csi-driver/blob/ef62ad26615e00b6bbcff901d5b967149855556f/deploy/csi-blob-driver.yaml#L13-L14

[andyzhangx]

wait, if you want to use workload identity, you should not use secretName in volume parameter, refer to https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/docs/workload-identity-static-pv-mount.md