search

Azure / azure-storage-fuse

A virtual file system adapter for Azure Blob storage
Other
647 stars 204 forks source link

Failure to mount storage account in pod when in Redhat Openshift (self managed) #1442

Closed jsanchezmartinez closed 3 weeks ago

jsanchezmartinez commented 3 weeks ago

### Which version of blobfuse was used? BLOBFUSE_VERSION: 1.4.5 BLOBFUSE2_VERSION: 2.3.0

Which OS distribution and version are you using?

OpenShift 4.14

NAME="Red Hat Enterprise Linux CoreOS" ID="rhcos" ID_LIKE="rhel fedora" VERSION="414.92.202405070148-0" VERSION_ID="4.14" VARIANT="CoreOS" VARIANT_ID=coreos PLATFORM_ID="platform:el9" PRETTY_NAME="Red Hat Enterprise Linux CoreOS 414.92.202405070148-0 (Plow)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:9::coreos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://docs.openshift.com/container-platform/4.14/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform" REDHAT_BUGZILLA_PRODUCT_VERSION="4.14" REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform" REDHAT_SUPPORT_PRODUCT_VERSION="4.14" OPENSHIFT_VERSION="4.14" RHEL_VERSION="9.2" OSTREE_VERSION="414.92.202405070148-0"

What was the issue encountered?

As indicated in https://github.com/kubernetes-sigs/blob-csi-driver/issues/1141, Pod shows failure in mounting the volume with log message [mkdir /.blobfuse2: operation not permitted] with fuse2 protocol when --default-working-dir mount option is not specified. When --default-working-dir=/tmp/blobfuse2 is specified as mount options then pod is able to mount the storage account without any error.

Is this something expected but not documented?

Have you found a mitigation/solution?

--default-working-dir=/tmp/blobfuse2 is specified as mount options.

Please share logs if available.

volumeId openshift-uat-euw#ocpuateuw#pvc-e9572eda-f387-4d84-bc00-362594df31ad##bcn-cps-apps# context map[containername:pvc-e9572eda-f387-4d84-bc00-362594df31ad csi.storage.k8s.io/pv/name:pvc-e9572eda-f387-4d84-bc00-362594df31ad csi.storage.k8s.io/pvc/name:blob csi.storage.k8s.io/pvc/namespace:bcn-cps-apps location:westeurope resourceGroup:openshift-uat-euw secretnamespace:bcn-cps-apps skuName:Standard_ZRS storage.kubernetes.io/csiProvisionerIdentity:1719823405129-5976-blob.csi.azure.com storageAccount:ocpuateuw storeAccountKey:false] mountflags [-o allow_other] mountOptions [-o allow_other -o gid=1000840000 --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/openshift-uat-euw#ocpuateuw#pvc-e9572eda-f387-4d84-bc00-362594df31ad##bcn-cps-apps# --container-name=pvc-e9572eda-f387-4d84-bc00-362594df31ad] volumeMountGroup 1000840000 args /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7674dc8983fe4b73ede24907bcc6791b073bd6655afbbf7daa14c7bc06bcf8f8/globalmount -o allow_other -o gid=1000840000 --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/openshift-uat-euw#ocpuateuw#pvc-e9572eda-f387-4d84-bc00-362594df31ad##bcn-cps-apps# --container-name=pvc-e9572eda-f387-4d84-bc00-362594df31ad serverAddress ocpuateuw.blob.core.windows.net I0701 09:40:16.723049 18732 nodeserver.go:166] start connecting to blobfuse proxy, protocol: , args: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7674dc8983fe4b73ede24907bcc6791b073bd6655afbbf7daa14c7bc06bcf8f8/globalmount -o allow_other -o gid=1000840000 --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/openshift-uat-euw#ocpuateuw#pvc-e9572eda-f387-4d84-bc00-362594df31ad##bcn-cps-apps# --container-name=pvc-e9572eda-f387-4d84-bc00-362594df31ad I0701 09:40:16.723635 18732 nodeserver.go:175] begin to mount with blobfuse proxy, protocol: , args: /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7674dc8983fe4b73ede24907bcc6791b073bd6655afbbf7daa14c7bc06bcf8f8/globalmount -o allow_other -o gid=1000840000 --pre-mount-validate=true --use-https=true --cancel-list-on-mount-seconds=10 --empty-dir-check=false --tmp-path=/mnt/openshift-uat-euw#ocpuateuw#pvc-e9572eda-f387-4d84-bc00-362594df31ad##bcn-cps-apps# --container-name=pvc-e9572eda-f387-4d84-bc00-362594df31ad E0701 09:40:16.735404 18732 nodeserver.go:178] GRPC call returned with an error:rpc error: code = Unknown desc = exit status 1 Error: failed to create default work dir [mkdir /.blobfuse2: operation not permitted] E0701 09:40:16.735457 18732 nodeserver.go:435] rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to create default work dir [mkdir /.blobfuse2: operation not permitted] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. I0701 09:40:16.735653 18732 azure_metrics.go:115] "Observed Request Latency" latency_seconds=1.539359679 request="blob_csi_driver_node_stage_volume" resource_group="openshift-uat-euw" subscription_id="" source="blob.csi.azure.com" volumeid="openshift-uat-euw#ocpuateuw#pvc-e9572eda-f387-4d84-bc00-362594df31ad##bcn-cps-apps#" result_code="failed_csi_driver_node_stage_volume" E0701 09:40:16.735687 18732 utils.go:109] GRPC error: rpc error: code = Internal desc = Mount failed with error: rpc error: code = Unknown desc = exit status 1 Error: failed to create default work dir [mkdir /.blobfuse2: operation not permitted] , output: Please refer to http://aka.ms/blobmounterror for possible causes and solutions for mount errors. I0701 09:40:17.319671 18732 utils.go:104] GRPC call: /csi.v1.Node/NodeStageVolume I0701 09:40:17.319689 18732 utils.go:105] GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7674dc8983fe4b73ede24907bcc6791b073bd6655afbbf7daa14c7bc06bcf8f8/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["-o allow_other"],"volume_mount_group":"1000840000"}},"access_mode":{"mode":5}},"volume_context":{"containername":"pvc-e9572eda-f387-4d84-bc00-362594df31ad","csi.storage.k8s.io/pv/name":"pvc-e9572eda-f387-4d84-bc00-362594df31ad","csi.storage.k8s.io/pvc/name":"blob","csi.storage.k8s.io/pvc/namespace":"bcn-cps-apps","location":"westeurope","resourceGroup":"openshift-uat-euw","secretnamespace":"bcn-cps-apps","skuName":"Standard_ZRS","storage.kubernetes.io/csiProvisionerIdentity":"1719823405129-5976-blob.csi.azure.com","storageAccount":"ocpuateuw","storeAccountKey":"false"},"volume_id":"openshift-uat-euw#ocpuateuw#pvc-e9572eda-f387-4d84-bc00-362594df31ad##bcn-cps-apps#"} I0701 09:40:17.321239 18732 blob.go:525] volumeID(openshift-uat-euw#ocpuateuw#pvc-e9572eda-f387-4d84-bc00-362594df31ad##bcn-cps-apps#) authEnv: [] I0701 09:40:17.340818 18732 blob.go:595] get account(ocpuateuw) key from secret(bcn-cps-apps, azure-storage-account-ocpuateuw-secret) failed with error: could not get secret(azure-storage-account-ocpuateuw-secret): secrets "azure-storage-account-ocpuateuw-secret" not found, use cluster identity to get account key instead I0701 09:40:17.626012 18732 nodeserver.go:394] append volumeMountGroup 1000840000 I0701 09:40:17.626049 18732 nodeserver.go:409] target /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/7674dc8983fe4b73ede24907bcc6791b073bd6655afbbf7daa14c7bc06bcf8f8/globalmount protocol

souravgupta-msft commented 3 weeks ago

Hi @jsanchezmartinez. The default value of the default-working-dir is $HOME/.blobfuse2. mkdir /.blobfuse2: operation not permitted In your case the $HOME environment variable is not set. So, it is being set to /.blobfuse2 where the process does not have required permissions. So, you can either set your $HOME env variable or provide the working directory using the cli param, which you have already tried out.

souravgupta-msft commented 3 weeks ago

The required documentation can be found here, --default-working-dir: The default working directory to store log files and other blobfuse2 related information.

jsanchezmartinez commented 3 weeks ago

Hi @souravgupta-msft, Do you mean at Pod level (I can't see that option in the Helm Chart)? Because in the Nodes, $HOME refers to "/root". Also checked in the Pods (through terminal), and $HOME value seems to be picked up correctly from Node. Regards,

souravgupta-msft commented 3 weeks ago

The logic is that if the $HOME environment variable is not set, the default working directory is set to current directory which is ./.blobfuse2. In your case it looks like the value of $HOME is "/" as a result of which the path is set as /.blobfuse2

jsanchezmartinez commented 3 weeks ago

Hi, The topic is that $HOME is in fact set:

# echo $HOME
/root

We'll probably have to add "--default-working-dir" to the StorageClass. Regards,

souravgupta-msft commented 3 weeks ago

Blobfuse is not reading the above value for $HOME as it can be seen in the error message. It might be some issue with the AKS environment. Even if $HOME is read correctly, you will still get the same permissions error trying to create a directory in "/root". So, using the CLI flag is the right option here.

Closing as the CLI flag is working. For more queries you can post in this thread.