Open torredil opened 6 months ago
@torredil I guess this is reason for my problem at https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1665#issuecomment-1974447831
Is this also the reason for https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1955 ?
@primeroz No, the root cause for #1955 was identified as a regression introduced by a breaking change in Karpenter related to taints: https://github.com/kubernetes-sigs/karpenter/pull/508 - that issue was fixed by https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/1969 and released in driver v1.29.0 (the corresponding add-on release is currently in progress and is expected to be available by March 29)
This specific issue was created to address a scenario / race condition where executing the prestop lifecycle hook is undesirable based on EC2 API behavior - today, DetachVolume
requests wait for the TerminateInstances
workflow to complete before proceeding. That is to say, if Karpenter has already called TerminateInstances
there is no benefit in executing the lifecycle hook.
The optimal solution here would be to make Karpenter aware of volume teardown before invoking TerminateInstances
as part of the node drain process. This approach would eliminate the need for executing a lifecycle hook in the first place and prevent delayed attachments as a result of termination events.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
/remove-lifecycle rotten
Issue Description
Currently, the CSI node pod utilizes OS graceful shutdown time during termination events by waiting for volumes on a node to be detached because Kubelet's node shutdown manager does not consider volume cleanup as a signal and tends to prematurely terminate the driver pod, which can result in delayed attachments.
Ideally,
DetachVolume
calls should complete beforeTerminateInstances
is invoked - however, in scenarios whereTerminateInstances
is called by autoscalers such as Karpenter, it is undesirable for the lifecycle hook to wait for detachments to succeed in the backend becauseDetachVolume
requests made afterTerminateInstances
will wait for theTerminateInstances
workflow to complete.Additional Context
According to the external attacher docs, checking for the presence of
DeletionTimestamp
in VA objects can be used to ensure that volumes have been successfully unmounted from k8s perspective (which is more reliable than scanning for mounts on the host).Also, kubelet sets a
NotReady
condition on the node with the reason set to "node is shutting down", which can be used to determine node status to stop waiting on detaches in specifically the scenario described above.