I dynamically provisioned a volume and then attached it to a deployment. When I delete a node and let Cluster API provision a new node, it might happen that dangling volumeattachments are left behind. In a future drain of a node, pods might get stuck in ContainerCreating state due to this.
k describe po -n kube-system nginx-59d9859785-lm97h
Name: nginx-59d9859785-lm97h
Namespace: kube-system
Priority: 0
Service Account: default
Node: kubermatic-v3-test-worker-57ccd5c88c-55n5f/10.70.27.38
Start Time: Mon, 11 Dec 2023 08:52:50 +0100
Labels: app=nginx
pod-template-hash=59d9859785
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/nginx-59d9859785
Containers:
nginx:
Container ID:
Image: nginx
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from csi-data-vcdplugin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dn4fn (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
csi-data-vcdplugin:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: csi-pvc-vcdplugin
ReadOnly: false
kube-api-access-dn4fn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 9m55s default-scheduler 0/5 nodes are available: 1 node(s) had untolerated taint {node.cloudprovider.kubernetes.io/uninitialized: true}, 1 node(s) were unschedulable, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling..
Normal Scheduled 4m58s default-scheduler Successfully assigned kube-system/nginx-59d9859785-lm97h to kubermatic-v3-test-worker-57ccd5c88c-55n5f
Warning FailedAttachVolume 4m59s attachdetach-controller Multi-Attach error for volume "pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2" Volume is already exclusively attached to one node and can't be attached to another
Warning FailedMount 70s kubelet Unable to attach or mount volumes: unmounted volumes=[csi-data-vcdplugin], unattached volumes=[csi-data-vcdplugin], failed to process volumes=[]: timed out waiting for the condition
Dangling volumeattachments:
k get volumeattachments.storage.k8s.io | grep pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2
csi-33c5c660443f4a1d8fa7ec048bfe010699b9245cc2f0d2aac30db6b3b665f600 named-disk.csi.cloud-director.vmware.com pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2 kubermatic-v3-test-worker-57ccd5c88c-65zg2 true 2d19h
csi-da9c3c68da884d8459778f4dfedd66cc28bcb7d1e0c89bb8abb967ce6d36407f named-disk.csi.cloud-director.vmware.com pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2 kubermatic-v3-test-worker-57ccd5c88c-54m5f true 2d19h
k apply -n default -f nginx.yaml
persistentvolumeclaim/csi-pvc-vcdplugin created
deployment.apps/nginx created
Once the pod is running, drain and delete a node, let Cluster API provision a new one.
k drain --ignore-daemonsets --delete-emptydir-data node/kubermatic-v3-test-worker-57ccd5c88c-54m5f
k delete machine.cluster.k8s.io/kubermatic-v3-test-worker-57ccd5c88c-54m5f
Drain a node.
k drain --ignore-daemonsets --delete-emptydir-data node/node/kubermatic-v3-test-worker-57ccd5c88c-65zg2
It may take a few attempts as it does not happen every time.
I can delete the ContainerCreating pod with the --force flag. But the pod still does not start.
k delete po -n kube-system nginx-59d9859785-lm97h --force
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "nginx-59d9859785-lm97h" force deleted
k describe po -n kube-system nginx-59d9859785-gbprn
Name: nginx-59d9859785-gbprn
Namespace: kube-system
Priority: 0
Service Account: default
Node: kubermatic-v3-test-worker-57ccd5c88c-55n5f/10.70.27.38
Start Time: Mon, 11 Dec 2023 09:23:56 +0100
Labels: app=nginx
pod-template-hash=59d9859785
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/nginx-59d9859785
Containers:
nginx:
Container ID:
Image: nginx
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from csi-data-vcdplugin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dwr9q (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
csi-data-vcdplugin:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: csi-pvc-vcdplugin
ReadOnly: false
kube-api-access-dwr9q:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13s default-scheduler Successfully assigned kube-system/nginx-59d9859785-gbprn to kubermatic-v3-test-worker-57ccd5c88c-55n5f
Warning FailedAttachVolume 14s attachdetach-controller Multi-Attach error for volume "pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2" Volume is already exclusively attached to one node and can't be attached to another
When I try to delete the previous volumeattachments:
Only the one on the just drained and still existing node is deleted. The one on the non existing node remains:
k describe volumeattachments.storage.k8s.io csi-da9c3c68da884d8459778f4dfedd66cc28bcb7d1e0c89bb8abb967ce6d36407f
Name: csi-da9c3c68da884d8459778f4dfedd66cc28bcb7d1e0c89bb8abb967ce6d36407f
Namespace:
Labels: <none>
Annotations: csi.alpha.kubernetes.io/node-id: kubermatic-v3-test-worker-57ccd5c88c-54m5f
API Version: storage.k8s.io/v1
Kind: VolumeAttachment
Metadata:
Creation Timestamp: 2023-12-08T12:48:14Z
Deletion Grace Period Seconds: 0
Deletion Timestamp: 2023-12-08T12:52:26Z
Finalizers:
external-attacher/named-disk-csi-cloud-director-vmware-com
Managed Fields:
API Version: storage.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:csi.alpha.kubernetes.io/node-id:
f:finalizers:
.:
v:"external-attacher/named-disk-csi-cloud-director-vmware-com":
Manager: csi-attacher
Operation: Update
Time: 2023-12-08T12:48:14Z
API Version: storage.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:attacher:
f:nodeName:
f:source:
f:persistentVolumeName:
Manager: kube-controller-manager
Operation: Update
Time: 2023-12-08T12:48:14Z
API Version: storage.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:attached:
f:attachmentMetadata:
.:
f:diskID:
f:diskUUID:
f:filesystem:
f:vmID:
f:detachError:
.:
f:message:
f:time:
Manager: csi-attacher
Operation: Update
Subresource: status
Time: 2023-12-11T08:34:24Z
Resource Version: 1992050
UID: 583797f0-5ebf-46c0-82f6-27c01478c085
Spec:
Attacher: named-disk.csi.cloud-director.vmware.com
Node Name: kubermatic-v3-test-worker-57ccd5c88c-54m5f
Source:
Persistent Volume Name: pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2
Status:
Attached: true
Attachment Metadata:
Disk ID: pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2
Disk UUID: 6000c295-355f-da02-a25a-f852b7ce31d8
Filesystem: ext4
Vm ID: kubermatic-v3-test-worker-57ccd5c88c-54m5f
Detach Error:
Message: rpc error: code = NotFound desc = Could not find VM with nodeID [kubermatic-v3-test-worker-57ccd5c88c-54m5f] from which to detach [pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2]
Time: 2023-12-11T08:34:24Z
Events: <none>
The new pod will start:
k describe po -n kube-system nginx-59d9859785-gbprn
Name: nginx-59d9859785-gbprn
Namespace: kube-system
Priority: 0
Service Account: default
Node: kubermatic-v3-test-worker-57ccd5c88c-55n5f/10.70.27.38
Start Time: Mon, 11 Dec 2023 09:23:56 +0100
Labels: app=nginx
pod-template-hash=59d9859785
Annotations: <none>
Status: Running
IP: 10.244.10.63
IPs:
IP: 10.244.10.63
Controlled By: ReplicaSet/nginx-59d9859785
Containers:
nginx:
Container ID: containerd://ed9a503d4249f0a9b837c73a0c7063b83b98e781599f0727755225ef900cd927
Image: nginx
Image ID: docker.io/library/nginx@sha256:10d1f5b58f74683ad34eb29287e07dab1e90f10af243f151bb50aa5dbb4d62ee
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Mon, 11 Dec 2023 09:30:42 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from csi-data-vcdplugin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dwr9q (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
csi-data-vcdplugin:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: csi-pvc-vcdplugin
ReadOnly: false
kube-api-access-dwr9q:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m18s default-scheduler Successfully assigned kube-system/nginx-59d9859785-gbprn to kubermatic-v3-test-worker-57ccd5c88c-55n5f
Warning FailedAttachVolume 7m18s attachdetach-controller Multi-Attach error for volume "pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2" Volume is already exclusively attached to one node and can't be attached to another
Warning FailedMount 3m (x2 over 5m15s) kubelet Unable to attach or mount volumes: unmounted volumes=[csi-data-vcdplugin], unattached volumes=[csi-data-vcdplugin], failed to process volumes=[]: timed out waiting for the condition
Normal SuccessfulAttachVolume 49s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2"
Normal Pulling 48s kubelet Pulling image "nginx"
Normal Pulled 32s kubelet Successfully pulled image "nginx" in 15.20183854s (15.20186002s including waiting)
Normal Created 32s kubelet Created container nginx
Normal Started 32s kubelet Started container nginx
A new volumeattachments is created, but the dangling remains:
k get volumeattachments.storage.k8s.io | grep pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2
csi-c53775fdb3196e08803afeddc59a6fb79f4e3a054241fbcd629ecd09a18b28af named-disk.csi.cloud-director.vmware.com pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2 kubermatic-v3-test-worker-57ccd5c88c-55n5f true 5m10s
csi-da9c3c68da884d8459778f4dfedd66cc28bcb7d1e0c89bb8abb967ce6d36407f named-disk.csi.cloud-director.vmware.com pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2 kubermatic-v3-test-worker-57ccd5c88c-54m5f true 2d19h
If I now drain again, I have the same issue again:
k describe po -n kube-system nginx-59d9859785-6k2nx
Name: nginx-59d9859785-6k2nx
Namespace: kube-system
Priority: 0
Service Account: default
Node: kubermatic-v3-test-worker-57ccd5c88c-65zg2/10.70.27.39
Start Time: Mon, 11 Dec 2023 09:36:39 +0100
Labels: app=nginx
pod-template-hash=59d9859785
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/nginx-59d9859785
Containers:
nginx:
Container ID:
Image: nginx
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from csi-data-vcdplugin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bqg5w (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
csi-data-vcdplugin:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: csi-pvc-vcdplugin
ReadOnly: false
kube-api-access-bqg5w:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m3s default-scheduler Successfully assigned kube-system/nginx-59d9859785-6k2nx to kubermatic-v3-test-worker-57ccd5c88c-65zg2
Warning FailedAttachVolume 3m4s attachdetach-controller Multi-Attach error for volume "pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2" Volume is already used by pod(s) nginx-59d9859785-gbprn
Warning FailedMount 61s kubelet Unable to attach or mount volumes: unmounted volumes=[csi-data-vcdplugin], unattached volumes=[csi-data-vcdplugin], failed to process volumes=[]: timed out waiting for the condition
Now when I remove the finalizers of the dangling volumeattachment:
Delete the volumeattachment of the just drained node:
k delete volumeattachments.storage.k8s.io csi-c53775fdb3196e08803afeddc59a6fb79f4e3a054241fbcd629ecd09a18b28af
volumeattachment.storage.k8s.io "csi-c53775fdb3196e08803afeddc59a6fb79f4e3a054241fbcd629ecd09a18b28af" deleted
A new volumeattachment is created:
k get volumeattachments.storage.k8s.io | grep pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2
csi-33c5c660443f4a1d8fa7ec048bfe010699b9245cc2f0d2aac30db6b3b665f600 named-disk.csi.cloud-director.vmware.com pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2 kubermatic-v3-test-worker-57ccd5c88c-65zg2 true 20s
And the pod is able to start:
k describe po -n kube-system nginx-59d9859785-6k2nx
Name: nginx-59d9859785-6k2nx
Namespace: kube-system
Priority: 0
Service Account: default
Node: kubermatic-v3-test-worker-57ccd5c88c-65zg2/10.70.27.39
Start Time: Mon, 11 Dec 2023 09:36:39 +0100
Labels: app=nginx
pod-template-hash=59d9859785
Annotations: <none>
Status: Running
IP: 10.244.9.246
IPs:
IP: 10.244.9.246
Controlled By: ReplicaSet/nginx-59d9859785
Containers:
nginx:
Container ID: containerd://d2467ccd11fa6a0524208e248eeabf2ceafd81e48bfa6e3c192a1c4528a1907a
Image: nginx
Image ID: docker.io/library/nginx@sha256:10d1f5b58f74683ad34eb29287e07dab1e90f10af243f151bb50aa5dbb4d62ee
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Mon, 11 Dec 2023 09:43:09 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/nginx/html from csi-data-vcdplugin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bqg5w (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
csi-data-vcdplugin:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: csi-pvc-vcdplugin
ReadOnly: false
kube-api-access-bqg5w:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m52s default-scheduler Successfully assigned kube-system/nginx-59d9859785-6k2nx to kubermatic-v3-test-worker-57ccd5c88c-65zg2
Warning FailedAttachVolume 6m53s attachdetach-controller Multi-Attach error for volume "pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2" Volume is already used by pod(s) nginx-59d9859785-gbprn
Warning FailedMount 2m36s (x2 over 4m50s) kubelet Unable to attach or mount volumes: unmounted volumes=[csi-data-vcdplugin], unattached volumes=[csi-data-vcdplugin], failed to process volumes=[]: timed out waiting for the condition
Normal SuccessfulAttachVolume 26s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-8032a622-19c0-4e66-aacf-c0911ac8bda2"
Normal Pulling 24s kubelet Pulling image "nginx"
Normal Pulled 23s kubelet Successfully pulled image "nginx" in 1.073543173s (1.073553964s including waiting)
Normal Created 23s kubelet Created container nginx
Normal Started 23s kubelet Started container nginx
When I now drain again, the pod is able to start without issues and manual intervention.
Describe the bug
I dynamically provisioned a volume and then attached it to a deployment. When I delete a node and let Cluster API provision a new node, it might happen that dangling volumeattachments are left behind. In a future drain of a node, pods might get stuck in ContainerCreating state due to this.
Dangling volumeattachments:
Reproduction steps
1. 2. 3. ...
Expected behavior
I expected the pod to successfully start on another node.
Additional context
Apply the manifest.
Once the pod is running, drain and delete a node, let Cluster API provision a new one.
Drain a node.
Verify you have dangling volumeattachments.
It may take a few attempts as it does not happen every time.
I can delete the ContainerCreating pod with the --force flag. But the pod still does not start.
When I try to delete the previous volumeattachments:
Only the one on the just drained and still existing node is deleted. The one on the non existing node remains:
The new pod will start:
A new volumeattachments is created, but the dangling remains:
If I now drain again, I have the same issue again:
Now when I remove the finalizers of the dangling volumeattachment:
Delete the volumeattachment of the just drained node:
A new volumeattachment is created:
And the pod is able to start:
When I now drain again, the pod is able to start without issues and manual intervention.
Kubernetes version:
Cloud provider VMware Cloud Director version: 10.4.2.21954589
OS version:
Install tools: KubeOne v1.7.1
Container runtime (CRI) and version (if applicable) containerd://1.6.25
Related plugins (CNI, CSI, ...) and versions (if applicable)
Logs: csi-vcd-nodeplugin-zg6b4.zip csi-vcd-controllerplugin-76cff99975-xh8vc.zip kubelet-journal-kubermatic-v3-test-worker-57ccd5c88c-65zg2.zip