search

kubernetes / cloud-provider-openstack

Apache License 2.0
619 stars 610 forks source link

[occm]when delete LB type of service, LB is already deleted when controller tries to delete it #1931

Closed suntech-lab closed 2 years ago

suntech-lab commented 2 years ago

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug /kind feature

What happened: When delete a loadbalancer type of service, "kubectl delete" hangs forever waiting for the service is deleted. The service is not deleted because controller failed to delete LB, which because the LB is already deleted. Here is a snippet of logging.

E0702 17:01:48.178423       1 controller.go:310] error processing service demo/svc-foo (will retry): failed to check if load balancer exists before cleanup: Internal Server Error
I0702 17:01:48.178546       1 event.go:294] "Event occurred" object="demo/svc-foo" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to check if load balancer exists before cleanup: Internal Server Error"
I0702 17:01:48.322537       1 event.go:294] "Event occurred" object="demo/svc-bar" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="DeletingLoadBalancer" message="Deleting load balancer"
I0702 17:01:48.443856       1 loadbalancer.go:3137] "Matching floating IP" floatingIP="10.91.242.87" description="Floating IP for Kubernetes external service demo/svc-bar from cluster kubernetes"
I0702 17:01:48.443907       1 loadbalancer.go:3144] "Deleting floating IP for service" floatingIP="10.91.242.87" service="demo/svc-bar"                                                                                                       
I0702 17:01:49.192877       1 loadbalancer.go:3150] "Deleted floating IP for service" floatingIP="10.91.242.87" service="demo/svc-bar"                                                                                                        
I0702 17:01:49.192903       1 loadbalancer.go:3162] "Deleting load balancer" lbID="33051ad8-0d7b-4b81-945a-eaf35e94548c" service="demo/svc-bar"
E0702 17:01:51.580674       1 controller.go:310] error processing service demo/svc-bar (will retry): failed to delete load balancer: Internal Server Error
I0702 17:01:51.580803       1 event.go:294] "Event occurred" object="demo/svc-bar" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to delete load balancer: Internal Server Error"

What you expected to happen: The service is deleted properly.

How to reproduce it: Delete a loadbalancer type of service.

Anything else we need to know?:

Environment:

jichenjc commented 2 years ago

controller failed to delete LB, which because the LB is already deleted.

are you sure the LB is already deleted? as we have handling for the not found issue

Internal Server Error might be the root cause , but I don't know where it comes out..how about with --v=6 to OCCM and show additional logs to see anything might be helpful

ialidzhikov commented 2 years ago

Does it also happen with 1.24.1? Because in 1.24.1 we cherry-picked a fix for https://github.com/kubernetes/cloud-provider-openstack/issues/1912 and it should not try to delete a loadbalancer when it does not exist anymore.

suntech-lab commented 2 years ago

After rebuilding the cluster, the issue is gone. Sorry for the false alarm.