[BUG] - CSI not able to mount volume to node

samson4649 commented 1 year ago

Describe the bug Worker node is not able to have a volume mounted. The PVC is in a ready state and the volume exists however the volumeattachments.storage.k8s.io object is reporting ATTACHED=false with the message rpc error: code = DeadlineExceeded desc = context deadline exceeded.

This has happened multiple times in the last 2 weeks with both a newly created volume and an existing volume after a work load shift between nodes.

To Reproduce Not exactly sure how to reproduce however the following is the description of the volumeattachment when the error occurs

Name:         csi-b1725867<REDACTED>
Namespace:    
Labels:       <none>
Annotations:  csi.alpha.kubernetes.io/node-id: d5b41b30-<REDACTED>
API Version:  storage.k8s.io/v1
Kind:         VolumeAttachment
Metadata:
  Creation Timestamp:  2022-11-18T09:47:38Z
  Finalizers:
    external-attacher/block-csi-vultr-com
  Managed Fields:
    API Version:  storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:csi.alpha.kubernetes.io/node-id:
        f:finalizers:
          .:
          v:"external-attacher/block-csi-vultr-com":
    Manager:      Go-http-client
    Operation:    Update
    Time:         2022-11-18T09:47:38Z
    API Version:  storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:attacher:
        f:nodeName:
        f:source:
          f:persistentVolumeName:
    Manager:      kube-controller-manager
    Operation:    Update
    Time:         2022-11-18T09:47:38Z
    API Version:  storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:attachError:
          .:
          f:message:
          f:time:
    Manager:         Go-http-client
    Operation:       Update
    Subresource:     status
    Time:            2022-11-18T22:18:15Z
  Resource Version:  11029444
  UID:               262caf1b-<REDACTED>
Spec:
  Attacher:   block.csi.vultr.com
  Node Name:  <REDACTED>
  Source:
    Persistent Volume Name:  pvc-<REDACTED>
Status:
  Attach Error:
    Message:  rpc error: code = DeadlineExceeded desc = context deadline exceeded
    Time:     2022-11-18T22:18:15Z
  Attached:   false
Events:       <none>

Expected behavior When a PVC has been defined and the PV is created, the volume should be able to be mounted to a pod without issues.

Screenshots

Desktop (please complete the following information where applicable:

OS - Vultr managed VKE cluster

samson4649 commented 1 year ago

After some additional debugging, the following was also noted

A reboot of the host did not fix the issue.
Scaling up, moving pods to another worker node, the scaling down did not fix the issue.
If only 11 volumes are mounted to a node, all volumes mount as expected.
if 12 (or more) volumes are scheduled to be mounted to a node, only 11 will successfully mount.

Nov 19 10:52:01 <REDACTED> kubelet[2945]: E1119 10:52:01.983018    2945 kubelet.go:1731] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[registry-data], unattached volumes=[registry-config registry-data registry-htpasswd]: timed out waiting for the condition" pod="<REDACTED>"
Nov 19 10:52:01 <REDACTED> kubelet[2945]: E1119 10:52:01.983858    2945 pod_workers.go:965] "Error syncing pod, skipping" err="unmounted volumes=[registry-data], unattached volumes=[registry-config registry-data registry-htpasswd]: timed out waiting for the condition" pod="<REDACTED>" podUID=e5738463-891c-4d6d-9cd1-4d54680f55cd

happytreees commented 1 year ago

Hello @samson4649

We found your cluster and the ticket you opened relating to this issue. After review it looks like you are hitting the block storage limits for an individual instance. The support staff will reach out on your ticket with more information.

Closing because this is not a bug.

vultr / vultr-csi

[BUG] - CSI not able to mount volume to node #99