search

kubernetes-sigs / karpenter

Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.
Apache License 2.0
627 stars 204 forks source link

Karpenter should show `Disrupting` or `Terminating` through `kubectl get nodes` when it has tainted Nodes #1152

Open jonathan-innis opened 7 months ago

jonathan-innis commented 7 months ago

Description

What problem are you trying to solve?

Currently, Kubernetes uses the node.kubernetes.io/unschedulable taint and the spec.unschedulable field on the node to mark that a node is cordoned and may be about to be drained for maintenance or removal. This is visible through the printer columns that you get when you call kubectl get nodes like the following

NAME                                                    STATUS                     ROLES    AGE    VERSION
ip-192-168-10-60.us-west-2.compute.internal             Ready                      <none>   2d4h   v1.28.5-eks-5e0fdde
ip-192-168-125-1.us-west-2.compute.internal             Ready                      <none>   2d4h   v1.28.5-eks-5e0fdde
ip-192-168-9-118.us-west-2.compute.internal             Ready,SchedulingDisabled   <none>   2d4h   v1.28.5-eks-5e0fdde

The code for this handling can be seen in the printer columns logic for kubectl here.

This is nice visibility for users when Kubernetes is using this specific field; however, nothing is surfaced when Karpenter adds its taint and is actively draining the node since Karpenter doesn't update the spec.unschedulable field that the printer relies on to add the SchedulingDisabled section to the node.

It would be a really nice UX if we could add something similar to SchedulingDisabled (perhaps something like Disrupting or Terminating) to the node so that users get visibility through the printer that Karpenter is acting on the node.

jonathan-innis commented 7 months ago

Also, as part of the alignment effort with Cluster Autoscaler, I imagine whatever change that we suggested should be made in upstream would also apply to Cluster Autoscaler. Perhaps aligning on the taint that we both want to use and proposing a way that that taint could have special logic built around it in the node printer columns is a change that we could try to get into upstream?

cc: @MaciekPytel @towca

jonathan-innis commented 7 months ago

Also also, there was a discussion in the K8s Slack over whether Karpenter should be using the unschedulable field on the node to piggy-back on this logic or not. Practically, we have chosen to separate ourselves from the basic "cordon" handling for the following reasons:

  1. We want to know that Karpenter was the one that initiated the tainting so that we can recover when we taint a node and the process gets killed (either due to crashing or upgrading)
  2. We want to make sure that we can drain DaemonSets while draining the node if the user wants it (right now Kubernetes will add a tolerations to DaemonSets for the node.kubernetes.io/unschedulable taint by default)
k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

simon-wessel commented 3 months ago

I would like to note that other controllers are relying on the SchedulingDisabled-Taint to see if a node is shutting down or not.

The CloudNativePG operator for example has a PDB that disallows deleting the pod. If a node with a CNPG pod receives the SchedulingDisabled-Taint, the operator will start to migrate that pod itself. Since Karpenter does not use the taint, the node is stuck.

simon-wessel commented 3 months ago

/remove-lifecycle stale

k8s-triage-robot commented 2 weeks ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

simon-wessel commented 2 weeks ago

/remove-lifecycle stale