search

kubernetes-sigs / cluster-api-provider-openstack

Cluster API implementation for OpenStack
https://cluster-api-openstack.sigs.k8s.io/
Apache License 2.0
298 stars 256 forks source link

kubernetes cluster upgrade breaks with single-node control plane #1425

Open scrungus opened 1 year ago

scrungus commented 1 year ago

/kind bug

What steps did you take and what happened: kubernetes cluster upgrade breaks with single-node control plane - either fails to swap the floating IP over to the new control node or unassigns the floating IP completely

Screenshot 2022-12-14 at 17 04 27

This is a kubernetes cluster with 2 worker nodes and 1 control node that was deployed via helm on a management cluster, using the openstack cluster api provider to spin up machines. It was deployed with kubernetes version v1.24.2. A helm upgrade was performed to v1.25.4. The controller successfully spun up a control node and two worker nodes with the new kubernetes version and deleted the old ones, but failed to swap over the floating IP to the new control node.

What did you expect to happen: the original control node should retain the floating IP until the new one is ready, then the floating IP should be moved over to the new control node

Anything else you would like to add: testing an upgrade from kubernetes v1.24.2 to v1.25.4

Environment:

jichenjc commented 1 year ago

I am wondering whether this is same to https://github.com/kubernetes-sigs/cluster-api-provider-openstack/issues/1265

as that issue also talk about 

The cause of the issue is quite straightforward: as we don't check if floatingIP is already associated to an healthy control-plane machine, each machine will [attach the floating IP to its port when it is reconcilated.](https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/v0.6.3/controllers/openstackmachine_controller.go#L384).

so which CAPO you are using?

scrungus commented 1 year ago

We are running v0.7.0-rc.0

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

mdbooth commented 1 year ago

It's probably too late to ask for logs on this one, but I wonder what the sequence of events here is. Presumably we're going to add the new control plane node first before taking the old one down. In that case I'd expect to see this log message:

https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/34ed24f3726d8b13c2b4a634cf96a739cf286e66/controllers/openstackmachine_controller.go#L412-L415

This doesn't return an error so won't cause another reconcile attempt. If it did return an error, it would likely prevent the old control plane node from being removed. My gut feeling is that this can't work.

I suspect we should move the API load balancer code out of the machine controller and into the cluster controller. Or possibly even its own controller 🤔

/remove-lifecycle stale

Ritikaa96 commented 1 year ago

/kind bug

k8s-triage-robot commented 10 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 9 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

mdbooth commented 9 months ago

/remove-lifecycle rotten

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

mdbooth commented 6 months ago

/remove-lifecycle stale

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

EmilienM commented 3 months ago

/remove-lifecycle stale

k8s-triage-robot commented 5 days ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale