previous node taint is wiped when kubelet restarts

Chaunceyctx commented 1 year ago

What happened?

I have a k8s cluster(v1.27.2) containing one node. I set evictionHard: nodefs.available: 90% and write a large amount of data to kubelet rootdir（used 8GB / total 10GB ） to trigger eviction. node.kubernetes.io/disk-pressure taint was added to this node. But when kubelet restarted, previous disk-pressure taint was wiped weirdly. And pending pod is normally scheduled to run on the current node. Then I checked the kubelet logs and found:

kubelet restart:

I0727 11:09:05.782733 3813426 flags.go:64] FLAG: --tls-private-key-file=""
I0727 11:09:05.782737 3813426 flags.go:64] FLAG: --topology-manager-policy="none"
I0727 11:09:05.782741 3813426 flags.go:64] FLAG: --topology-manager-scope="container"
I0727 11:09:05.782745 3813426 flags.go:64] FLAG: --v="8"
I0727 11:09:05.782749 3813426 flags.go:64] FLAG: --version="false"
I0727 11:09:05.782754 3813426 flags.go:64] FLAG: --vmodule=""
I0727 11:09:05.782757 3813426 flags.go:64] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
I0727 11:09:05.782761 3813426 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
I0727 11:09:05.796690 3813426 mount_linux.go:222] Detected OS with systemd

update node status NodeHasNoDiskPressure

I0727 11:09:10.858628 3813426 kubelet_node_status.go:632] "Recording event message for node" node="192.168.2.107" event="NodeHasNoDiskPressure"
I0727 11:09:10.858640 3813426 kubelet_node_status.go:762] "Setting node status condition code" position=9 node="192.168.2.107"

eviction manager start to synchronize

I0727 11:09:13.373393 3813426 eviction_manager.go:292] "Eviction manager: synchronize housekeeping"

Q: Why does kubelet report NodeHasNoDiskPressure ? A: Eviction manager has not yet executed the synchronize method

Normally, a node that was previously marked with taints should retain after reboot otherwise it doesn't make sense. Maybe we need to make sure that taints can't be deleted when kubelet restarts?

What did you expect to happen?

previous node taint is not wiped

How can we reproduce it (as minimally and precisely as possible)?

Restart kubelet repeatedly after the disk pressure eviction is triggered. Observe node.spec.taints

Anything else we need to know?

No response

Kubernetes version

1.27.2

Cloud provider

OS version

```console # On Linux: $ cat /etc/os-release # paste output here $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ```

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

Chaunceyctx commented 1 year ago

/sig node /assign

Chaunceyctx commented 1 year ago

@bobbypage @liggitt @tallclair @Random-Liu PTAL

ffromani commented 1 year ago

/cc

tzneal commented 1 year ago

/triage accepted

Chaunceyctx commented 1 year ago

@klueska @dashpole @liggitt @derekwaynecarr @pacoxu PTAL. Thanks!

heartlock commented 1 year ago

I have encountered the same bug, is there a plan to fix it?

SuQiucheng commented 1 year ago

I hope the community can fix this bug as soon as possible.

waiterQ commented 1 year ago

I also met this before, is there a fix plan?

Chaunceyctx commented 6 months ago

@ffromani @tzneal I believe that the impact of this problem is actually very small, and the taints of the node will disappear briefly and then appear again. Therefore, I would like to ask if it is still necessary for us to repair this problem

kubernetes / kubernetes