"install-blobfuse-proxy" container fails on OpenShift (error copying blobfuse2 - "Text file busy")

[calohmn]

What happened: We are using blob-csi-driver-v1.23.2 on OpenShift. We are frequently seeing csi-blob-node pods to not get ready because of an error in the install-blobfuse-proxy container. The error is about copying the blobfuse2 executable: cp: cannot create regular file '/host/usr/local/bin/blobfuse2': Text file busy

Full log:

+ INSTALL_BLOBFUSE_PROXY=true
+ DISABLE_UPDATEDB=true
+ SET_MAX_OPEN_FILE_NUM=true
+ SET_READ_AHEAD_SIZE=true
+ READ_AHEAD_KB=15380
+ HOST_CMD=nsenter --mount=/proc/1/ns/mnt
+ nsenter --mount=/proc/1/ns/mnt cat /etc/os-release
+ grep ^ID=
+ cut -d= -f2
+ tr -d "
+ DISTRIBUTION=rhcos
+ nsenter --mount=/proc/1/ns/mnt uname -m
+ ARCH=x86_64
+ echo Linux distribution: rhcos, Arch: x86_64
Linux distribution: rhcos, Arch: x86_64
+ . ./blobfuse-proxy/install-proxy-rhcos.sh
+ set -xe
+ [ true = true ]
+ echo copy blobfuse2....
+ copy blobfuse2....
cp /usr/bin/blobfuse2 /host/usr/local/bin/blobfuse2
cp: cannot create regular file '/host/usr/local/bin/blobfuse2': Text file busy

It looks like a fix similar to #776 needs to be applied to pkg/blobfuse-proxy/install-proxy-rhcos.sh, deleting /host/usr/local/bin/blobfuse2 first before copying the blobfuse2 executable: https://github.com/kubernetes-sigs/blob-csi-driver/blob/7e434dbc816a4a390dd6f58390e7c82cfa8d4e63/pkg/blobfuse-proxy/install-proxy-rhcos.sh#L19-L24

What you expected to happen: install-blobfuse-proxy container startup without error.

How to reproduce it: Using the blob-csi-driver on OpenShift and restarting a csi-blob-node pod.

Anything else we need to know?:

Environment:

• CSI Driver version: 1.23.2
• Kubernetes version (use kubectl version): v1.25.11+1485cc9
• OS (e.g. from /etc/os-release): rhcos
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[andyzhangx]

/assign @cvvz

[andyzhangx]

can you set INSTALL_BLOBFUSE2 as false in the blob csi driver daemonset? the error would exist if you try to install blobfuse2 during csi driver start up.

[calohmn]

@andyzhangx I see that this would circumvent the issue, yes. But this would mean that I have to have the intended blobfuse2 version already installed on the node beforehand, wouldn't it? I would rather use the default of having the blob-csi-driver handle the blobfuse2 installation, being able to configure the intended version via the blob-csi-driver helm chart values.

How about only copying the blobfuse2 executable if it doesn't exist yet in the target directory or if the checksum differs (like it is done with the blobfuse-proxy executable). This would cover the case that blobfuse2 already got installed via a csi-blob-node pod but then the pod got restarted for some reason, triggering the installation again.