search

dell / csi-baremetal

Bare-metal CSI Driver
Apache License 2.0
73 stars 33 forks source link

Kubelet spams NodeUnstageVolume calls #908

Open yurkov-anton opened 2 years ago

yurkov-anton commented 2 years ago

Describe the bug If deleted volume by force, without success NodeUnstageVolume, then log contain follow spam:

Dec 13 23:17:28.354 [INFO] [CRHelper] [GetVolumeByID] [pvc-6211b936-656c-494a-be7a-28c9846c4108] Volume CR not found
Dec 13 23:17:28.354 [ERRO] [CSINodeService] [NodeUnstageVolume] [pvc-6211b936-656c-494a-be7a-28c9846c4108] Unable to find volume with ID pvc-6211b936-656c-494a-be7a-28c9846c4108
Dec 13 23:17:28.376 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-ce222eeb-3fc4-4241-8baa-379b2c9fce52] locking volume on request: volume_id:"pvc-ce222eeb-3fc4-4241-8baa-379b2c9fce52" staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-ce222eeb-3fc4-4241-8baa-379b2c9fce52/globalmount"
Dec 13 23:17:28.387 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-e99a3945-b77b-4cb5-bb23-6607227dc497] locking volume on request: volume_id:"pvc-e99a3945-b77b-4cb5-bb23-6607227dc497" staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e99a3945-b77b-4cb5-bb23-6607227dc497/globalmount"
Dec 13 23:17:28.390 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-5968be9f-9eb8-416c-9fa3-837a125684da] locking volume on request: volume_id:"pvc-5968be9f-9eb8-416c-9fa3-837a125684da" staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5968be9f-9eb8-416c-9fa3-837a125684da/globalmount"
Dec 13 23:17:28.390 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-5968be9f-9eb8-416c-9fa3-837a125684da] Processing request
Dec 13 23:17:28.404 [INFO] [CRHelper] [GetVolumeByID] [pvc-91c12620-baf4-4c99-81ad-ea26fe2fc770] Volume CR not found
Dec 13 23:17:28.404 [ERRO] [CSINodeService] [NodeUnstageVolume] [pvc-91c12620-baf4-4c99-81ad-ea26fe2fc770] Unable to find volume with ID pvc-91c12620-baf4-4c99-81ad-ea26fe2fc770
Dec 13 23:17:28.453 [INFO] [CRHelper] [GetVolumeByID] [pvc-16bc8535-b667-4a58-9b3f-8cf74a22e6a6] Volume CR not found
Dec 13 23:17:28.453 [ERRO] [CSINodeService] [NodeUnstageVolume] [pvc-16bc8535-b667-4a58-9b3f-8cf74a22e6a6] Unable to find volume with ID pvc-16bc8535-b667-4a58-9b3f-8cf74a22e6a6
Dec 13 23:17:28.453 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-18b02415-6640-4aad-8605-f9ea92e7487f] Processing request
Dec 13 23:17:28.504 [INFO] [CRHelper] [GetVolumeByID] [pvc-78aedd46-6bc1-4a1b-a4c7-62c254d3597e] Volume CR not found
Dec 13 23:17:28.504 [ERRO] [CSINodeService] [NodeUnstageVolume] [pvc-78aedd46-6bc1-4a1b-a4c7-62c254d3597e] Unable to find volume with ID pvc-78aedd46-6bc1-4a1b-a4c7-62c254d3597e
Dec 13 23:17:28.504 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-4ce6408d-b637-4c04-af99-9f6ede767e74] Processing request
Dec 13 23:17:28.549 [INFO] [CSINodeService] [NodeUnstageVolume] [pvc-e8529962-4836-4874-a9bd-289fad2311c3] locking volume on request: volume_id:"pvc-e8529962-4836-4874-a9bd-289fad2311c3" staging_target_path:"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e8529962-4836-4874-a9bd-289fad2311c3/globalmount"

This spam leads to an increase in the size of the csi and kubelet logs. (10mb for 10 minutes for ~40 error volumes)

If volume was removed without successful NodeUnstageVolume, kubelet will try to call NodeUnstageVolume, and CSI return NOT_FOUND.

Environment (please complete the following information):

To Reproduce

  1. Create volume.
  2. Delete volume by patch.
  3. Delete sts, pod and pv

Expected behavior kubelet log and csi node log will contain spam with NodeUnstageVolume calls.

Additional context Possible hotfix spam is restart kubelet.

yurkov-anton commented 2 years ago

Need to cherry-pick to release-1.1 branch