search

Azure / kubernetes-volume-drivers

Kubernetes volume drivers for Azure
MIT License
126 stars 55 forks source link

[local volume provisioner]temp disk may be corrupted if it's used by two drivers or applications. #117

Open andyzhangx opened 2 years ago

andyzhangx commented 2 years ago

What happened:

Since for the temp disk, it may be used by other applications or driver, e.g. blob csi driver, if both drivers or applications are writing data to /dev/sdb1 temp disk, that disk would be corrupted easily.

Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    22s               default-scheduler  Successfully assigned default/deployment-localdisk-6f95f4f858-w4srx to aks-nodepool1-40942288-vmss000003
  Warning  FailedMount  6s (x6 over 22s)  kubelet            MountVolume.MountDevice failed for volume "local-pv-4b838508" : local: failed to mount device /dev/sdb1 at /var/lib/kubelet/plugins/kubernetes.io/local-volume/mounts/local-pv-4b838508 (fstype: ext4), error format of disk "/dev/sdb1" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/local-volume/mounts/local-pv-4b838508") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.44.1 (24-Mar-2018)
/dev/sdb1 is mounted; will not make a filesystem here!
)

Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    19s               default-scheduler  Successfully assigned default/deployment-localdisk-6f95f4f858-zmbkd to aks-nodepool1-40942288-vmss000003
  Warning  FailedMount  4s (x6 over 19s)  kubelet            MountVolume.MountDevice failed for volume "local-pv-4b838508" : local: failed to mount device /dev/sdb1 at /var/lib/kubelet/plugins/kubernetes.io/local-volume/mounts/local-pv-4b838508 (fstype: ext4), error format of disk "/dev/sdb1" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/local-volume/mounts/local-pv-4b838508") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.44.1 (24-Mar-2018)
/dev/sdb1 is apparently in use by the system; will not make a filesystem here!
)

# echo a > b
bash: b: Structure needs cleaning

workaround

run umount /mnt on the agent node

What you expected to happen:

How to reproduce it:

Anything else we need to know?:

Environment:

andyzhangx commented 2 years ago

Option#2: discover temp(/dev/sdb1) disk

to make sure temp disk is not used, run following command to unmount temp disk first

kubectl apply -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/local/umount-mnt.yaml
kubectl apply -f https://raw.githubusercontent.com/Azure/kubernetes-volume-drivers/master/local/local-pv-provisioner-tempdisk.yaml