search

kubernetes-sigs / aws-ebs-csi-driver

CSI driver for Amazon EBS https://aws.amazon.com/ebs/
Apache License 2.0
966 stars 779 forks source link

Optimize LCH Behavior During Termination Events #1926

Open torredil opened 6 months ago

torredil commented 6 months ago

Issue Description

Currently, the CSI node pod utilizes OS graceful shutdown time during termination events by waiting for volumes on a node to be detached because Kubelet's node shutdown manager does not consider volume cleanup as a signal and tends to prematurely terminate the driver pod, which can result in delayed attachments.

Ideally, DetachVolume calls should complete before TerminateInstances is invoked - however, in scenarios where TerminateInstances is called by autoscalers such as Karpenter, it is undesirable for the lifecycle hook to wait for detachments to succeed in the backend because DetachVolume requests made after TerminateInstances will wait for the TerminateInstances workflow to complete.

Additional Context

According to the external attacher docs, checking for the presence of DeletionTimestamp in VA objects can be used to ensure that volumes have been successfully unmounted from k8s perspective (which is more reliable than scanning for mounts on the host).

Also, kubelet sets a NotReady condition on the node with the reason set to "node is shutting down", which can be used to determine node status to stop waiting on detaches in specifically the scenario described above.

levanlongktmt commented 6 months ago

@torredil I guess this is reason for my problem at https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1665#issuecomment-1974447831

primeroz commented 5 months ago

Is this also the reason for https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1955 ?

torredil commented 5 months ago

@primeroz No, the root cause for #1955 was identified as a regression introduced by a breaking change in Karpenter related to taints: https://github.com/kubernetes-sigs/karpenter/pull/508 - that issue was fixed by https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/1969 and released in driver v1.29.0 (the corresponding add-on release is currently in progress and is expected to be available by March 29)

This specific issue was created to address a scenario / race condition where executing the prestop lifecycle hook is undesirable based on EC2 API behavior - today, DetachVolume requests wait for the TerminateInstances workflow to complete before proceeding. That is to say, if Karpenter has already called TerminateInstances there is no benefit in executing the lifecycle hook.

The optimal solution here would be to make Karpenter aware of volume teardown before invoking TerminateInstances as part of the node drain process. This approach would eliminate the need for executing a lifecycle hook in the first place and prevent delayed attachments as a result of termination events.

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

AndrewSirenko commented 1 month ago

/remove-lifecycle rotten