CLM could wait forever for node decommissioning - Githubissues

zalando-incubator / cluster-lifecycle-manager

Cluster Lifecycle Manager (CLM) to provision and update multiple Kubernetes clusters

MIT License

230 stars 24 forks source link

CLM could wait forever for node decommissioning #533

Open AlexanderYastrebov opened 2 years ago

AlexanderYastrebov commented 2 years ago

CLM could wait forever for node decommissioning (if there is some kind of problem)

https://github.com/zalando-incubator/cluster-lifecycle-manager/blob/769565ccaa59ba839fc127a0b907dd1687311617/pkg/updatestrategy/clc_update.go#L48-L89

https://github.com/zalando-incubator/cluster-lifecycle-manager/blob/769565ccaa59ba839fc127a0b907dd1687311617/pkg/updatestrategy/clc_update.go#L109-L140 and therefore does not update other node pools.

See also https://github.com/zalando-incubator/cluster-lifecycle-manager/issues/154

AlexanderYastrebov commented 2 years ago

Some ideas

use child context.WithTimeout in the PrepareForRemoval and doUpdate
do cluster config update before nodepool updates https://github.com/zalando-incubator/cluster-lifecycle-manager/blob/769565ccaa59ba839fc127a0b907dd1687311617/provisioner/clusterpy.go#L424
do not return early on pool update error https://github.com/zalando-incubator/cluster-lifecycle-manager/blob/769565ccaa59ba839fc127a0b907dd1687311617/provisioner/clusterpy.go#L400-L409 https://github.com/zalando-incubator/cluster-lifecycle-manager/blob/769565ccaa59ba839fc127a0b907dd1687311617/provisioner/node_pools.go#L206-L219