CephFS: ceph-fuse: when cephfs nodeplugin pod restart, mount point cannot ls

[VictoryAnn]

Describe the bug

A clear and concise description of what the bug is.

Environment details

• Image/version of Ceph CSI driver : v3.6.2
• Helm chart version : 3-canary
• Kernel version : 4.14.105-19-0021 SMP Tue Aug 17 10:49:46 CST 2021 x86_64 x86_64 x86_64 GNU/Linux
• Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its krbd or rbd-nbd) : fuse
• Kubernetes cluster version : 1.21
• Ceph cluster version : 16.2.6

Steps to reproduce

Steps to reproduce the behavior:

1. create pvc https://github.com/ceph/ceph-csi/blob/devel/examples/cephfs/pvc.yaml
2. create deployment https://github.com/ceph/ceph-csi/blob/devel/examples/cephfs/deployment.yaml
3. restart all cephfs nodeplugin pods
4. enter demo pod and ls mountPath (ls /var/lib/www/html)

Actual results

Transport endpoint is not connected

ps -ef | grep ceph-fuse found ceph-fuse process not exists

Expected behavior

when cephfs-nodeplugin container or pod restart , my demo pod access normal mounPath.

Logs

If the issue is in PVC creation, deletion, cloning please attach complete logs of below containers.

• csi-provisioner and csi-rbdplugin/csi-cephfsplugin container logs from the provisioner pod.

If the issue is in PVC resize please attach complete logs of below containers.

• csi-resizer and csi-rbdplugin/csi-cephfsplugin container logs from the provisioner pod.

If the issue is in snapshot creation and deletion please attach complete logs of below containers.

• csi-snapshotter and csi-rbdplugin/csi-cephfsplugin container logs from the provisioner pod.

If the issue is in PVC mounting please attach complete logs of below containers.

• csi-rbdplugin/csi-cephfsplugin and driver-registrar container logs from plugin pod from the node where the mount is failing.

• if required attach dmesg logs.

Note:- If its a rbd issue please provide only rbd related logs, if its a cephFS issue please provide cephFS logs.

Additional context

ps -ef | grep ceph-fuse ceph-fuse process not exists rbd-nbd when nodeplugin restart , it will heal rbd-nbd process. cephfs also need to heal when restart or upgrade

For example:

Any existing bug report which describe about the similar issue/behavior

[Rakshith-R]

@VictoryAnn This is a known issue, please refer to https://github.com/ceph/ceph-csi/blob/devel/docs/ceph-mount-corruption.md. It requires manual steps, it cannot be handled within Cephcsi.