search

kubernetes / mount-utils

Package mount defines an interface to mounting filesystems.
Apache License 2.0
62 stars 33 forks source link

fix: set "host is down" as corrupted mount #3

Closed andyzhangx closed 3 years ago

andyzhangx commented 3 years ago

What type of PR is this? /kind bug

What this PR does / why we need it: fix: set "host is down" as corrupted mount

When SMB server is down, there is no way to terminate pod which is using SMB mount, would get following error. This PR regard host is down as corrupted mount dir, and then I would work out another PR in k8s to skip UnmountVolume.TearDown process if host is down

Apr 20 11:11:52 aks-nonzone-17963928-vmss000000 kubelet[8516]: E0420 11:11:52.618206    8516 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/smb.csi.k8s.io^pvc-da830ab0-e8d2-4e0e-89f4-906a4e4398a4 podName:90ee5993-7f80-4fe3-9d88-7bc89b19cbfe nodeName:}" failed. No retries permitted until 2021-04-20 11:11:56.61815874 +0000 UTC m=+337.879942743 (durationBeforeRetry 4s). Error: "UnmountVolume.TearDown failed for volume \"persistent-storage\" (UniqueName: \"kubernetes.io/csi/smb.csi.k8s.io^pvc-da830ab0-e8d2-4e0e-89f4-906a4e4398a4\") pod \"90ee5993-7f80-4fe3-9d88-7bc89b19cbfe\" (UID: \"90ee5993-7f80-4fe3-9d88-7bc89b19cbfe\") : kubernetes.io/csi: mounter.TearDownAt failed to clean mount dir [/var/lib/kubelet/pods/90ee5993-7f80-4fe3-9d88-7bc89b19cbfe/volumes/kubernetes.io~csi/pvc-da830ab0-e8d2-4e0e-89f4-906a4e4398a4/mount]: stat /var/lib/kubelet/pods/90ee5993-7f80-4fe3-9d88-7bc89b19cbfe/volumes/kubernetes.io~csi/pvc-da830ab0-e8d2-4e0e-89f4-906a4e4398a4/mount: host is down"

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Release note:

fix: set "host is down" as corrupted mount
k8s-ci-robot commented 3 years ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: andyzhangx To complete the pull request process, please assign saad-ali after the PR has been reviewed. You can assign the PR to them by writing /assign @saad-ali in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/kubernetes/mount-utils/blob/master/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
andyzhangx commented 3 years ago

/assign @msau42 I see PR template of this repo is

Sorry, we do not accept changes directly against this repository. Please see
CONTRIBUTING.md for information on where and how to contribute instead.

Shall we use another contribution welcome PR template?

msau42 commented 3 years ago

Sorry, this is really confusing. But basically this repo is a mirror of kubernetes/kubernetes/staging/mount-utils. So the PR you have for kubernetes/kubernetes will get published here whenever a kubernetes tag is made.

andyzhangx commented 3 years ago

Sorry, this is really confusing. But basically this repo is a mirror of kubernetes/kubernetes/staging/mount-utils. So the PR you have for kubernetes/kubernetes will get published here whenever a kubernetes tag is made.

@msau42 in that case, there would be a dependency on k8s release, that's time consuming. I would still think SMB CSI driver could depend on https://github.com/kubernetes/utils/, it's more efficient to make some changes. Can we merge this PR into k8s/utils first? https://github.com/kubernetes/utils/pull/203

In the meantime, I will make same code change into k/k

msau42 commented 3 years ago

Intermediate kubernetes tags are also pushed too. So I don't think it should be a long wait.

andyzhangx commented 3 years ago

https://github.com/kubernetes/kubernetes/pull/101398 would fix this issue.