search

spinkube / runtime-class-manager

A Kubernetes operator to manage Runtime Classes
Apache License 2.0
24 stars 8 forks source link

Support for persistent node labels for node stop/restart #222

Open Mossaka opened 1 day ago

Mossaka commented 1 day ago

Context: In AKS clusters, when the cluster is stopped and restarted, new nodes are provisioned as part of the VMSS design. This leads to the loss of Kubernetes annotations on the nodes (i.e. kwasm.sh/kwasm-node=true), which impacts configurations that rely on those annotations. The current behavior of runtime-class-manager is to mutate containerd configurations based on annotations that are manually applied to nodes. However, since annotations are not persistent across cluster restarts, this causes issues with retaining the necessary configuration.

ref: https://github.com/spinkube/azure/issues/24

Proposal:

Labels such as kwasm.sh/kwasm-node=true should be used instead of annotations for the runtime-class-manager to identify and mutate node configurations. Unlike annotations, labels are persistent across cluster restarts and node reallocation.

The runtime-class-manager should be enhanced to detect and reapply configurations based on node labels automatically, without requiring manual intervention. This would ensure configurations persist across node restarts.

Lastly, the legacy code in node_controller.go needs to be cleaned up. cc @vdice @voigt

vdice commented 1 day ago
  1. rcm does utilize labels today (see job_controller.go). (The legacy kwasm-operator project uses annotations.) a. However, re: AKS node turnover: I just tested and am not currently seeing any labels I've applied to one node persist to the next that has taken its place after an AKS stop/start cycle. This seems like a blocker for the purposes of this ticket, right? My mistake: for nodes to inherit labels on AKS, they need to be applied to the nodepool via https://learn.microsoft.com/en-us/azure/aks/use-labels#create-a-node-pool-with-a-label. Thanks @Mossaka!
  2. I'd be happy to pick the node_controller.go work up if I can get clarity on the intended functionality. I assume its intention is to watch for new nodes and see if they have relevant labels (eg <shim name>=true) and if so install the corresponding shim(s)? So, this in combination with 1a means we should have successful app restarts during node turnover?