unable to rolling update cluster

[zetaab]

/kind bug

1. What kops version are you running? The command kops version, will display this information.

master

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

1.23.5

3. What cloud provider are you using?

openstack

4. What commands did you run? What is the simplest way to reproduce this issue?

kops rolling-update cluster --yes

5. What happened after the commands executed?

I can see following in logs

Error: failed to drain node "nodes-x-qyvcrz": error deregistering instance "d54ecc5d-3e45-4b48-8a5a-009e544c03f1", node "nodes-x-qyvcrz": failed to deregister instance from load balancers: failed to update pool membership: Resource not found: [PUT https://xxxx:13876/v2.0/lbaas/pools/de6f98e8-f3a1-4ddc-9f96-bc52aad81c2a/members/794c4950-2a15-495f-bc78-2a5ae41b2132], error message: {"faultcode": "Client", "faultstring": "Member 794c4950-2a15-495f-bc78-2a5ae41b2132 not found.", "debuginfo": null}

The problem is here https://github.com/kubernetes/kops/blob/master/upup/pkg/fi/cloudup/openstack/instance.go#L235-L253

row 235 lists the members but for some reason the member does not exist anymore in row 244. Possible solution is that we just return true, nil https://github.com/kubernetes/kops/blob/703422d584b210509517d724c2cca37ddb4b846a/upup/pkg/fi/cloudup/openstack/loadbalancer.go#L339-L340 in case of member does not exists

[zetaab]

also seeing

node "nodes-x-zbeba9": failed to deregister instance from load balancers: failed to update pool membership: Expected HTTP response code [200 201 202] when accessing [PUT https://x:13876/v2.0/lbaas/pools/fabb57d4-afd3-432e-b734-3cba65e2d69d/members/1a9784a4-776b-4b79-94e5-8bde9b3d9b42], but got 409 instead {"faultcode": "Client", "faultstring": "Pool fabb57d4-afd3-432e-b734-3cba65e2d69d is immutable and cannot be updated.", "debuginfo": null}