NEGs are not deleted during service deletion

[marianhromiak]

I'm having issue with the autoneg v0.9.9. Everything seems to be working fine since I deleted service with autoneg annotations.

I can see deregistering message in logs

DEBUG   events  Normal  {"object": {"kind":"Service","namespace":"test","name":"test-marianh","uid":"d362ac9e-f5dd-48ee-a167-91abf6c1be0a","apiVersion":"v1","resourceVersion":"14606392"}, "reason": "Delete", "message": "Deregistered NEGs for \"test/test-marianh\" from backend service \"autoneg-w1\" (port 80)"}"

but these NEGs will never be deleted or detached from backend service.

$ gcloud compute backend-services list --project=anthos-mesh-sandbox-1b827f9b 
NAME        BACKENDS                                                                                                                                                                                                                                            PROTOCOL
autoneg-w1  europe-west1-b/networkEndpointGroups/k8s1-6b097e81-test-test-marianh-80-1330420d,europe-west1-c/networkEndpointGroups/k8s1-6b097e81-test-test-marianh-80-1330420d,europe-west1-d/networkEndpointGroups/k8s1-6b097e81-test-test-marianh-80-1330420d  TCP

$ gcloud compute network-endpoint-groups list --project=anthos-mesh-sandbox-1b827f9b  | grep marianh
k8s1-6b097e81-test-test-marianh-80-1330420d                      europe-west1-b  GCE_VM_IP_PORT  0
k8s1-6b097e81-test-test-marianh-80-1330420d                      europe-west1-c  GCE_VM_IP_PORT  0
k8s1-6b097e81-test-test-marianh-80-1330420d                      europe-west1-d  GCE_VM_IP_PORT  1

Is there anything what can I check why these NEGs persist service deletion, or is there cook book how to correct delete service with NEG annotation?

[rosmo]

The NEGs are created by the cloud.google.com/neg annotation by GKE itself, Autoneg just adds these to a backend service. So it's more of an GKE issue I'd say...

[rosmo]

Ah, sorry, I did not see that these were not getting detached - let me have a look at this.

[ykyr]

Hi @rosmo, Do you know if there is any progress? 🙏 We switched to the autoneg-controller from the custom-controlled NEGs attachment, which works great. We have a very dynamic environment, and the cleanup after the deletion process is painful. We would expect the controller to handle it all end-to-end. Thanks!

[noaml-rb]

Hi, We have also encountered this issue. would love to know if this is worked on. Thanks 🙏

[rosmo]

I'll try to find some time soon, but always happy to receive patches!

[rosmo]

Can you give the new 0.9.10 release a try?

[idanyreblaze]

seems to work correctly in 0.9.10, thanks