Installation Failure due to Blobfuse Proxy Container

[Kaushik-Vaibhav]

What happened: I'm encountering an issue while setting up the Azure Blob CSI driver with the install-blobfuse-proxy container. It seems the installation script fails because it tries to reference a package that does not exist within the container, specifically /blobfuse-proxy/packages-microsoft-prod-18.04.deb

What you expected to happen: The installation script should complete without errors, and Blobfuse/Blobfuse2 should be installed properly in the container.

How to reproduce it:

1. Install csi driver using helm chart using

helm install blob-csi-driver blob-csi-driver/blob-csi-driver --set node.enableBlobfuseProxy=true --set controller.replicas=1 --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet" --namespace kube-system --version v1.25.0

2. Check for logs for install-blobfuse-proxy container kubectl logs pod/csi-blob-node-k6fvw -n kube-system -c install-blobfuse-proxy

delphix@vk-hs:~/hyperscale-poc$ kubectl logs pod/csi-blob-node-k6fvw -n kube-system -c install-blobfuse-proxy

+ INSTALL_BLOBFUSE_PROXY=true
+ DISABLE_UPDATEDB=true
+ SET_MAX_OPEN_FILE_NUM=true
+ SET_READ_AHEAD_SIZE=true
+ READ_AHEAD_KB=15380
+ KUBELET_PATH=/var/snap/microk8s/common/var/lib/kubelet
+ [ /var/snap/microk8s/common/var/lib/kubelet != /var/lib/kubelet ]
+ echo kubelet path is /var/snap/microk8s/common/var/lib/kubelet, update blobfuse-proxy.service....
+ sed -i s#--blobfuse-proxy-endpoint[^ ]*#--blobfuse-proxy-endpoint=unix://var/snap/microk8s/common/var/lib/kubelet/plugins/blob.csi.azure.com/blobfuse-proxy.sock# /blobfuse-proxy/blobfuse-proxy.service
kubelet path is /var/snap/microk8s/common/var/lib/kubelet, update blobfuse-proxy.service....
+ echo blobfuse-proxy endpoint is updated to unix://var/snap/microk8s/common/var/lib/kubelet/plugins/blob.csi.azure.com/blobfuse-proxy.sock
+ HOST_CMD=nsenter --mount=/proc/1/ns/mnt
blobfuse-proxy endpoint is updated to unix://var/snap/microk8s/common/var/lib/kubelet/plugins/blob.csi.azure.com/blobfuse-proxy.sock
+ nsenter --mount=/proc/1/ns/mnt cat /etc/os-release
+ grep ^ID=
+ + cut -d= -f2
tr -d "
+ DISTRIBUTION=ubuntu
+ nsenter --mount=/proc/1/ns/mnt uname -m
Linux distribution: ubuntu, Arch: x86_64
+ ARCH=x86_64
+ echo Linux distribution: ubuntu, Arch: x86_64
+ . ./blobfuse-proxy/install-proxy.sh
+ set -xe
+ [ ubuntu != ubuntu ]
+ [ true = true ]
+ echo start to install blobfuse/blobfuse2....
start to install blobfuse/blobfuse2....
+ nsenter --mount=/proc/1/ns/mnt lsb_release -rs
Ubuntu release: 20.04
+ release=20.04
+ echo Ubuntu release: 20.04
+ expr 20.04 < 22.04
+ [ 1 -eq 1 ]
+ cp /blobfuse-proxy/packages-microsoft-prod-18.04.deb /host/etc/packages-microsoft-prod.deb
cp: cannot stat '/blobfuse-proxy/packages-microsoft-prod-18.04.deb': No such file or directory

Anything else we need to know?: The same error also occurred when on setting '--set node.enableBlobfuseProxy=false'.

However, after uninstalling the current veriion, installing with an older version seemed to work helm install blob-csi-driver blob-csi-driver/blob-csi-driver --set node.enableBlobfuseProxy=true --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet" --set controller.replicas=1 --namespace kube-system --version v1.24.3

On checking the events of the container:

Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Normal   Scheduled         28s                default-scheduler  Successfully assigned kube-system/csi-blob-node-k6fvw to vk-hs
  Normal   Created           16s (x2 over 24s)  kubelet            Created container install-blobfuse-proxy
  Normal   Started           16s (x2 over 24s)  kubelet            Started container install-blobfuse-proxy
  Warning  BackOff           14s (x2 over 15s)  kubelet            Back-off restarting failed container install-blobfuse-proxy in pod csi-blob-node-k6fvw_kube-system(d3e83bd2-bb18-411c-9d26-d596f63814f4)
  Warning  DNSConfigForming  3s (x6 over 28s)   kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 172.16.101.11 172.16.105.2 8.8.8.8
  Normal   Pulled            3s (x3 over 28s)   kubelet            Container image "mcr.microsoft.com/oss/kubernetes-csi/blob-csi:v1.25.0" already present on machine

Environment: CSI Driver version: v1.25.0 Kubernetes version (use kubectl version): v1.27.4 OS (e.g. from /etc/os-release): "Ubuntu", (20.04.6 LTS) (Focal Fossa)" Kernel (e.g. uname -a): Linux vk-azure 5.15.0-1073-dx2024092516-650d167a5-azure https://github.com/kubernetes-sigs/blob-csi-driver/pull/82~20.04.1 SMP Wed Sep 25 16:38:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Install tools: snap (for microk8 installation)

[rainerleber]

@Kaushik-Vaibhav I have the same problem but fixed here: https://github.com/kubernetes-sigs/blob-csi-driver/pull/1651

[andyzhangx]

@Kaushik-Vaibhav this PR(https://github.com/kubernetes-sigs/blob-csi-driver/pull/1660) should fix the issue. In the meantime, if your node has already installed blobfuse2 package, you could set INSTALL_BLOBFUSE2 env var as false: https://github.com/kubernetes-sigs/blob-csi-driver/blob/6c830004f066da7d521ae9e0f23179daca1a111e/deploy/csi-blob-node.yaml#L61C25-L62

Or you just install the blobfuse2 package manually on the node as a workaround if it's not installed by default.