search

giantswarm / roadmap

Giant Swarm Product Roadmap
https://github.com/orgs/giantswarm/projects/273
Apache License 2.0
3 stars 0 forks source link

Clusterdeletion (via App) doesn't delete aadpodidentity.k8s.io/v1 resources #2206

Open bavarianbidi opened 1 year ago

bavarianbidi commented 1 year ago

Motivation

Deletion of a cluster (by deleting the App) doesn't cleanup the azureidentities and azureidentitiesbindings CRs:

on glippy:

k get azureidentities -A                                                                                                                                                                                                                 
NAMESPACE    NAME                                             TYPE   CLIENTID                               AGE                                                                                                                            
giantswarm   cluster01-org-multi-project-cluster-identity            0e1164c2-2809-4569-b557-d36b80ce2a83   9d                                                                                                                             
giantswarm   glippy-org-giantswarm-cluster-identity                  0e1164c2-2809-4569-b557-d36b80ce2a83   29d                                                                                                                            
giantswarm   mario01-org-multi-project-cluster-identity              0e1164c2-2809-4569-b557-d36b80ce2a83   17h                                                                                                                            
giantswarm   mario0815-org-multi-project-cluster-identity            0e1164c2-2809-4569-b557-d36b80ce2a83   2d16h                                                                                                                          
giantswarm   mariopriv01-org-multi-project-cluster-identity          0e1164c2-2809-4569-b557-d36b80ce2a83   19h                                                                                                                            
giantswarm   mariopriv02-org-multi-project-cluster-identity          0e1164c2-2809-4569-b557-d36b80ce2a83   19h                                                                                                                            
giantswarm   mcpriv01-org-multi-project-cluster-identity             0e1164c2-2809-4569-b557-d36b80ce2a83   53m
giantswarm   mcpriv02-org-multi-project-cluster-identity             0e1164c2-2809-4569-b557-d36b80ce2a83   44m
giantswarm   mcpub01-org-multi-project-cluster-identity              0e1164c2-2809-4569-b557-d36b80ce2a83   58m
giantswarm   mcpub02-org-multi-project-cluster-identity              0e1164c2-2809-4569-b557-d36b80ce2a83   43m
giantswarm   vpatest-org-multi-project-cluster-identity              0e1164c2-2809-4569-b557-d36b80ce2a83   16h

 k get azureidentitybindings -A
NAMESPACE    NAME                                                     AZUREIDENTITY                                    SELECTOR                                  AGE
giantswarm   cluster01-org-multi-project-cluster-identity-binding     cluster01-org-multi-project-cluster-identity     capz-controller-aadpodidentity-selector   9d
giantswarm   glippy-org-giantswarm-cluster-identity-binding           glippy-org-giantswarm-cluster-identity           capz-controller-aadpodidentity-selector   29d
giantswarm   mario01-org-multi-project-cluster-identity-binding       mario01-org-multi-project-cluster-identity       capz-controller-aadpodidentity-selector   17h
giantswarm   mario0815-org-multi-project-cluster-identity-binding     mario0815-org-multi-project-cluster-identity     capz-controller-aadpodidentity-selector   2d16h
giantswarm   mariopriv01-org-multi-project-cluster-identity-binding   mariopriv01-org-multi-project-cluster-identity   capz-controller-aadpodidentity-selector   19h
giantswarm   mariopriv02-org-multi-project-cluster-identity-binding   mariopriv02-org-multi-project-cluster-identity   capz-controller-aadpodidentity-selector   19h
giantswarm   mcpriv01-org-multi-project-cluster-identity-binding      mcpriv01-org-multi-project-cluster-identity      capz-controller-aadpodidentity-selector   54m
giantswarm   mcpriv02-org-multi-project-cluster-identity-binding      mcpriv02-org-multi-project-cluster-identity      capz-controller-aadpodidentity-selector   45m
giantswarm   mcpub01-org-multi-project-cluster-identity-binding       mcpub01-org-multi-project-cluster-identity       capz-controller-aadpodidentity-selector   59m
giantswarm   mcpub02-org-multi-project-cluster-identity-binding       mcpub02-org-multi-project-cluster-identity       capz-controller-aadpodidentity-selector   44m
giantswarm   vpatest-org-multi-project-cluster-identity-binding       vpatest-org-multi-project-cluster-identity       capz-controller-aadpodidentity-selector   16h

TODO

Outcome

For Customer

For Clippy

primeroz commented 1 year ago

those should get deleted by NMI i guess ( since it creates them )

Interestingly you can see that other than

all the others are very recent. you don't see any nikXXXX or fctestXXXX so i think they are getting deleted eventually

bavarianbidi commented 1 year ago

check the resources in the next couple of weeks to verify if they got cleaned up

bavarianbidi commented 1 year ago

Number of azureidentities and azureidentitybindings raised during the last month.

It seems that the deletion of a cluster (triggered by happa and in CI) differs from how we currently delete clusters when we do manual testing.

$ k get azureidentities -A
NAMESPACE    NAME                                             TYPE   CLIENTID                               AGE
giantswarm   bzm29-org-multi-project-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   27d
giantswarm   glippy-org-giantswarm-cluster-identity                  0e1164c2-2809-4569-b557-d36b80ce2a83   69d
giantswarm   jrp45-org-multi-project-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   28d
giantswarm   t-1q6ahgy-org-t-cgu1ijp-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   4d21h
giantswarm   t-2dm2l55-org-t-4johfw8-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   27d
giantswarm   t-32crglr-org-t-ubzrelz-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   5d21h
giantswarm   t-365re23-org-t-5dpxbqo-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   4d21h
giantswarm   t-4jut8qm-org-t-2ps97s4-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   25d
giantswarm   t-4n4zm6f-org-t-q277zgu-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   27d
giantswarm   t-5079rkq-org-t-8ap6ffp-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   18d
giantswarm   t-5x54m57-org-t-s0f73am-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   25d
giantswarm   t-65q82aq-org-t-geydx2l-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   13d
giantswarm   t-7h8267y-org-t-kksbkz5-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   17d
giantswarm   t-7va7t1y-org-t-soofzre-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   4d1h
giantswarm   t-85jt195-org-t-fzpfp5p-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   7d1h
giantswarm   t-9up521q-org-t-cnw2ime-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   13d
giantswarm   t-a7x12x3-org-t-nh6pa0b-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   4d
giantswarm   t-arf0wko-org-t-ta6hjwy-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   27d
giantswarm   t-c7cp63r-org-t-0xisjdd-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   20d
giantswarm   t-dogzzol-org-t-8ercp46-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   21d
giantswarm   t-f06pts5-org-t-fsiizrb-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   26d
giantswarm   t-f24bccg-org-t-ulw17xi-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   7d1h
giantswarm   t-fu9dxrt-org-t-gpdss9l-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   5d18h
giantswarm   t-gchjsvz-org-t-op3iglo-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   28d
giantswarm   t-kuxuyue-org-t-pgcpi4f-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   36d
giantswarm   t-m9bxpla-org-t-umf07sf-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   25d
giantswarm   t-mesutpd-org-t-sakj810-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   11d
giantswarm   t-mkemagm-org-t-4cmwc8v-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   28d
giantswarm   t-mr1b7lr-org-t-t270c0r-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   19d
giantswarm   t-nn433jq-org-t-w5rgdg2-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   4d22h
giantswarm   t-p6je3tb-org-t-mo9p1j3-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   10d
giantswarm   t-p9rlvmv-org-t-f21en87-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   25d
giantswarm   t-pb5tpwz-org-t-vs2p6ob-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   25d
giantswarm   t-pey4r33-org-t-bbqi9rt-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   4d22h
giantswarm   t-pj1kinf-org-t-gfgufa2-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   28d
giantswarm   t-r9xte96-org-t-xhums45-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   4d1h
giantswarm   t-rlkaxa4-org-t-kjd7w3h-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   20d
giantswarm   t-uh6qruf-org-t-5en1rsu-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   5d18h
giantswarm   t-uuzl7tf-org-t-fzc54wv-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   18d
giantswarm   t-uv4kxq4-org-t-5lce7lk-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   20d
giantswarm   t-vjw3rl8-org-t-ncl9y9k-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   25d
giantswarm   t-w5saxoa-org-t-fx2g392-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   26d
giantswarm   t-w7zw9cb-org-t-mnlijpz-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   4d21h
giantswarm   t-wss9rna-org-t-734bm0r-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   20d
giantswarm   t-x27k8iz-org-t-5fylfrv-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   25d
giantswarm   t-x3ebv1i-org-t-x5nstn8-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   20d
giantswarm   t-ye8yywv-org-t-940l9ss-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   4d
giantswarm   t-yiqxkbg-org-t-ofchwpe-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   28d
giantswarm   t-yznwei9-org-t-0scbmcj-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   26d
giantswarm   t-z3z34xn-org-t-mgagxb7-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   16d
giantswarm   t-zryehue-org-t-cisxbee-cluster-identity                0e1164c2-2809-4569-b557-d36b80ce2a83   5d22h
giantswarm   vpatest-org-multi-project-cluster-identity              0e1164c2-2809-4569-b557-d36b80ce2a83   40d
$ k get azureidentitybindings -A
NAMESPACE    NAME                                                     AZUREIDENTITY                                    SELECTOR                                  AGE
giantswarm   bzm29-org-multi-project-cluster-identity-binding         bzm29-org-multi-project-cluster-identity         capz-controller-aadpodidentity-selector   27d
giantswarm   glippy-org-giantswarm-cluster-identity-binding           glippy-org-giantswarm-cluster-identity           capz-controller-aadpodidentity-selector   69d
giantswarm   jrp45-org-multi-project-cluster-identity-binding         jrp45-org-multi-project-cluster-identity         capz-controller-aadpodidentity-selector   28d
giantswarm   t-1q6ahgy-org-t-cgu1ijp-cluster-identity-binding         t-1q6ahgy-org-t-cgu1ijp-cluster-identity         capz-controller-aadpodidentity-selector   4d21h
giantswarm   t-2dm2l55-org-t-4johfw8-cluster-identity-binding         t-2dm2l55-org-t-4johfw8-cluster-identity         capz-controller-aadpodidentity-selector   27d
giantswarm   t-32crglr-org-t-ubzrelz-cluster-identity-binding         t-32crglr-org-t-ubzrelz-cluster-identity         capz-controller-aadpodidentity-selector   5d21h
giantswarm   t-365re23-org-t-5dpxbqo-cluster-identity-binding         t-365re23-org-t-5dpxbqo-cluster-identity         capz-controller-aadpodidentity-selector   4d21h
giantswarm   t-4jut8qm-org-t-2ps97s4-cluster-identity-binding         t-4jut8qm-org-t-2ps97s4-cluster-identity         capz-controller-aadpodidentity-selector   25d
giantswarm   t-4n4zm6f-org-t-q277zgu-cluster-identity-binding         t-4n4zm6f-org-t-q277zgu-cluster-identity         capz-controller-aadpodidentity-selector   27d
giantswarm   t-5079rkq-org-t-8ap6ffp-cluster-identity-binding         t-5079rkq-org-t-8ap6ffp-cluster-identity         capz-controller-aadpodidentity-selector   18d
giantswarm   t-5x54m57-org-t-s0f73am-cluster-identity-binding         t-5x54m57-org-t-s0f73am-cluster-identity         capz-controller-aadpodidentity-selector   25d
giantswarm   t-65q82aq-org-t-geydx2l-cluster-identity-binding         t-65q82aq-org-t-geydx2l-cluster-identity         capz-controller-aadpodidentity-selector   13d
giantswarm   t-7h8267y-org-t-kksbkz5-cluster-identity-binding         t-7h8267y-org-t-kksbkz5-cluster-identity         capz-controller-aadpodidentity-selector   17d
giantswarm   t-7va7t1y-org-t-soofzre-cluster-identity-binding         t-7va7t1y-org-t-soofzre-cluster-identity         capz-controller-aadpodidentity-selector   4d1h
giantswarm   t-85jt195-org-t-fzpfp5p-cluster-identity-binding         t-85jt195-org-t-fzpfp5p-cluster-identity         capz-controller-aadpodidentity-selector   7d1h
giantswarm   t-9up521q-org-t-cnw2ime-cluster-identity-binding         t-9up521q-org-t-cnw2ime-cluster-identity         capz-controller-aadpodidentity-selector   13d
giantswarm   t-a7x12x3-org-t-nh6pa0b-cluster-identity-binding         t-a7x12x3-org-t-nh6pa0b-cluster-identity         capz-controller-aadpodidentity-selector   4d
giantswarm   t-arf0wko-org-t-ta6hjwy-cluster-identity-binding         t-arf0wko-org-t-ta6hjwy-cluster-identity         capz-controller-aadpodidentity-selector   27d
giantswarm   t-c7cp63r-org-t-0xisjdd-cluster-identity-binding         t-c7cp63r-org-t-0xisjdd-cluster-identity         capz-controller-aadpodidentity-selector   20d
giantswarm   t-dogzzol-org-t-8ercp46-cluster-identity-binding         t-dogzzol-org-t-8ercp46-cluster-identity         capz-controller-aadpodidentity-selector   21d
giantswarm   t-f06pts5-org-t-fsiizrb-cluster-identity-binding         t-f06pts5-org-t-fsiizrb-cluster-identity         capz-controller-aadpodidentity-selector   26d
giantswarm   t-f24bccg-org-t-ulw17xi-cluster-identity-binding         t-f24bccg-org-t-ulw17xi-cluster-identity         capz-controller-aadpodidentity-selector   7d1h
giantswarm   t-fu9dxrt-org-t-gpdss9l-cluster-identity-binding         t-fu9dxrt-org-t-gpdss9l-cluster-identity         capz-controller-aadpodidentity-selector   5d18h
giantswarm   t-gchjsvz-org-t-op3iglo-cluster-identity-binding         t-gchjsvz-org-t-op3iglo-cluster-identity         capz-controller-aadpodidentity-selector   28d
giantswarm   t-kuxuyue-org-t-pgcpi4f-cluster-identity-binding         t-kuxuyue-org-t-pgcpi4f-cluster-identity         capz-controller-aadpodidentity-selector   36d
giantswarm   t-m9bxpla-org-t-umf07sf-cluster-identity-binding         t-m9bxpla-org-t-umf07sf-cluster-identity         capz-controller-aadpodidentity-selector   25d
giantswarm   t-mesutpd-org-t-sakj810-cluster-identity-binding         t-mesutpd-org-t-sakj810-cluster-identity         capz-controller-aadpodidentity-selector   11d
giantswarm   t-mkemagm-org-t-4cmwc8v-cluster-identity-binding         t-mkemagm-org-t-4cmwc8v-cluster-identity         capz-controller-aadpodidentity-selector   28d
giantswarm   t-mr1b7lr-org-t-t270c0r-cluster-identity-binding         t-mr1b7lr-org-t-t270c0r-cluster-identity         capz-controller-aadpodidentity-selector   19d
giantswarm   t-nn433jq-org-t-w5rgdg2-cluster-identity-binding         t-nn433jq-org-t-w5rgdg2-cluster-identity         capz-controller-aadpodidentity-selector   4d22h
giantswarm   t-p6je3tb-org-t-mo9p1j3-cluster-identity-binding         t-p6je3tb-org-t-mo9p1j3-cluster-identity         capz-controller-aadpodidentity-selector   10d
giantswarm   t-p9rlvmv-org-t-f21en87-cluster-identity-binding         t-p9rlvmv-org-t-f21en87-cluster-identity         capz-controller-aadpodidentity-selector   25d
giantswarm   t-pb5tpwz-org-t-vs2p6ob-cluster-identity-binding         t-pb5tpwz-org-t-vs2p6ob-cluster-identity         capz-controller-aadpodidentity-selector   25d
giantswarm   t-pey4r33-org-t-bbqi9rt-cluster-identity-binding         t-pey4r33-org-t-bbqi9rt-cluster-identity         capz-controller-aadpodidentity-selector   4d22h
giantswarm   t-pj1kinf-org-t-gfgufa2-cluster-identity-binding         t-pj1kinf-org-t-gfgufa2-cluster-identity         capz-controller-aadpodidentity-selector   28d
giantswarm   t-r9xte96-org-t-xhums45-cluster-identity-binding         t-r9xte96-org-t-xhums45-cluster-identity         capz-controller-aadpodidentity-selector   4d1h
giantswarm   t-rlkaxa4-org-t-kjd7w3h-cluster-identity-binding         t-rlkaxa4-org-t-kjd7w3h-cluster-identity         capz-controller-aadpodidentity-selector   20d
giantswarm   t-uh6qruf-org-t-5en1rsu-cluster-identity-binding         t-uh6qruf-org-t-5en1rsu-cluster-identity         capz-controller-aadpodidentity-selector   5d18h
giantswarm   t-uuzl7tf-org-t-fzc54wv-cluster-identity-binding         t-uuzl7tf-org-t-fzc54wv-cluster-identity         capz-controller-aadpodidentity-selector   18d
giantswarm   t-uv4kxq4-org-t-5lce7lk-cluster-identity-binding         t-uv4kxq4-org-t-5lce7lk-cluster-identity         capz-controller-aadpodidentity-selector   20d
giantswarm   t-vjw3rl8-org-t-ncl9y9k-cluster-identity-binding         t-vjw3rl8-org-t-ncl9y9k-cluster-identity         capz-controller-aadpodidentity-selector   25d
giantswarm   t-w5saxoa-org-t-fx2g392-cluster-identity-binding         t-w5saxoa-org-t-fx2g392-cluster-identity         capz-controller-aadpodidentity-selector   26d
giantswarm   t-w7zw9cb-org-t-mnlijpz-cluster-identity-binding         t-w7zw9cb-org-t-mnlijpz-cluster-identity         capz-controller-aadpodidentity-selector   4d21h
giantswarm   t-wss9rna-org-t-734bm0r-cluster-identity-binding         t-wss9rna-org-t-734bm0r-cluster-identity         capz-controller-aadpodidentity-selector   20d
giantswarm   t-x27k8iz-org-t-5fylfrv-cluster-identity-binding         t-x27k8iz-org-t-5fylfrv-cluster-identity         capz-controller-aadpodidentity-selector   25d
giantswarm   t-x3ebv1i-org-t-x5nstn8-cluster-identity-binding         t-x3ebv1i-org-t-x5nstn8-cluster-identity         capz-controller-aadpodidentity-selector   20d
giantswarm   t-ye8yywv-org-t-940l9ss-cluster-identity-binding         t-ye8yywv-org-t-940l9ss-cluster-identity         capz-controller-aadpodidentity-selector   4d
giantswarm   t-yiqxkbg-org-t-ofchwpe-cluster-identity-binding         t-yiqxkbg-org-t-ofchwpe-cluster-identity         capz-controller-aadpodidentity-selector   28d
giantswarm   t-yznwei9-org-t-0scbmcj-cluster-identity-binding         t-yznwei9-org-t-0scbmcj-cluster-identity         capz-controller-aadpodidentity-selector   26d
giantswarm   t-z3z34xn-org-t-mgagxb7-cluster-identity-binding         t-z3z34xn-org-t-mgagxb7-cluster-identity         capz-controller-aadpodidentity-selector   16d
giantswarm   t-zryehue-org-t-cisxbee-cluster-identity-binding         t-zryehue-org-t-cisxbee-cluster-identity         capz-controller-aadpodidentity-selector   5d22h
giantswarm   vpatest-org-multi-project-cluster-identity-binding       vpatest-org-multi-project-cluster-identity       capz-controller-aadpodidentity-selector   40d
primeroz commented 1 year ago

was looking at the app we install

so what's creating those resources ?

primeroz commented 1 year ago

the identitybindings are actually created by capz-controller

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"10471dc3-32e0-4caf-8e29-155aad5f4e49","stage":"ResponseComplete","requestURI":"/apis/aadpodidentity.k8s.io/v1/namespaces/giantswarm/azureidentitybindings","verb":"create","user":{"username":"system:serviceaccount:giantswarm:capz-manager","uid":"805eb44d-96e0-496e-8e62-07742d601d15","groups":["system:serviceaccounts","system:serviceaccounts:giantswarm","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["capz-controller-manager-57865d9d75-lglwr"],"authentication.kubernetes.io/pod-uid":["c806b832-b2f5-441d-9034-ab924e22f40c"]}},"sourceIPs":["192.168.10.150"],"userAgent":"cluster-api-provider-azure-manager","objectRef":{"resource":"azureidentitybindings","namespace":"giantswarm","name":"glippy-org-giantswarm-cluster-identity-binding","apiGroup":"aadpodidentity.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","message":"azureidentitybindings.aadpodidentity.k8s.io \"glippy-org-giantswarm-cluster-identity-binding\" already exists","reason":"AlreadyExists","details":{"name":"glippy-org-giantswarm-cluster-identity-binding","group":"aadpodidentity.k8s.io","kind":"azureidentitybindings"},"code":409},"requestReceivedTimestamp":"2023-05-02T09:16:00.051077Z","stageTimestamp":"2023-05-02T09:16:00.062947Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"capz-aad-pod-id-nmi-binding\" of ClusterRole \"capz-aad-pod-id-nmi-role\" to ServiceAccount \"capz-manager/giantswarm\""}}

but i can also see create events from dns-operator-azure 

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"cd2e4fe6-fff7-4c50-b8d4-a51a223fa752","stage":"ResponseComplete","requestURI":"/apis/aadpodidentity.k8s.io/v1/namespaces/giantswarm/azureidentitybindings","verb":"create","user":{"username":"system:serviceaccount:giantswarm:dns-operator-azure","uid":"b5be9842-59d5-4733-9727-2bdc83bc7470","groups":["system:serviceaccounts","system:serviceaccounts:giantswarm","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["dns-operator-azure-8459dfff44-cm2d7"],"authentication.kubernetes.io/pod-uid":["96720666-a116-42a0-810f-cdcbc551dcf1"]}},"sourceIPs":["10.223.0.11"],"userAgent":"dns-operator-azure","objectRef":{"resource":"azureidentitybindings","namespace":"giantswarm","name":"t-pey4r33-org-t-bbqi9rt-cluster-identity-binding","apiGroup":"aadpodidentity.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","message":"azureidentitybindings.aadpodidentity.k8s.io \"t-pey4r33-org-t-bbqi9rt-cluster-identity-binding\" already exists","reason":"AlreadyExists","details":{"name":"t-pey4r33-org-t-bbqi9rt-cluster-identity-binding","group":"aadpodidentity.k8s.io","kind":"azureidentitybindings"},"code":409},"requestReceivedTimestamp":"2023-05-02T09:18:34.752716Z","stageTimestamp":"2023-05-02T09:18:34.802458Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"dns-operator-azure\" of ClusterRole \"dns-operator-azure\" to ServiceAccount \"dns-operator-azure/giantswarm\""}}

i cannot find any delete event for the resources, only the deletecollection from the namespace-controller

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"342de714-d0c9-45c4-a0da-0f4d46e1bcda","stage":"ResponseComplete","requestURI":"/apis/aadpodidentity.k8s.io/v1/namespaces/org-t-8ap6ffp/azureidentitybindings","verb":"deletecollection","user":{"username":"system:serviceaccount:kube-system:namespace-controller","uid":"d2b2c739-0550-4433-8e5c-69ae43f6bf89","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"sourceIPs":["10.223.0.133"],"userAgent":"kube-controller-manager/v1.24.11 (linux/amd64) kubernetes/0f75679/system:serviceaccount:kube-system:namespace-controller","objectRef":{"resource":"azureidentitybindings","namespace":"org-t-8ap6ffp","apiGroup":"aadpodidentity.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestObject":{"kind":"DeleteOptions","apiVersion":"meta.k8s.io/__internal","propagationPolicy":"Background"},"requestReceivedTimestamp":"2023-05-02T09:21:21.777952Z","stageTimestamp":"2023-05-02T09:21:21.779785Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:controller:namespace-controller\" of ClusterRole \"system:controller:namespace-controller\" to ServiceAccount \"namespace-controller/kube-system\""}}
primeroz commented 1 year ago

Looking at the code of capz controller https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/472d76774951f3d203aa9497b7a83fd1aad49548/controllers/azureidentity_controller.go#L162

        // only delete bindings when the identity owner type is not found.
        // we should not delete an identity when azureCluster is not found because it could have been created by AzureManagedControlPlane.

but no ownership is set on those identities still hanging around ( i checked a few ) ... looking at the switch case that will cause the identity not to be added to the list of identities to be deleted

primeroz commented 1 year ago

when looking at logs for t-zryehue-org-t-cisxbee-cluster-identity-binding 

➜ k get azureidentitybinding -n giantswarm t-zryehue-org-t-cisxbee-cluster-identity-binding 
NAME                                               AZUREIDENTITY                              SELECTOR                                  AGE
t-zryehue-org-t-cisxbee-cluster-identity-binding   t-zryehue-org-t-cisxbee-cluster-identity   capz-controller-aadpodidentity-selector   6d

the only event is 

 λ logs query '{cluster_id="glippy", app="capz-controller-manager"} |= "AzureIdentity" != "Watch close" != "Listing" != "Starting" != "Shutdown" != "Workers" |= "zryehue" ' --since 200h

2023-04-26T09:38:12Z {} I0426 09:38:12.958554       1 azureidentity_controller.go:133] controllers.AzureIdentityReconciler.Reconcile "msg"="object was not found" "AzureCluster"={"name":"t-zryehue","namespace":"org-t-cisxbee"} "controller"="AzureIdentity" "controllerGroup"="infrastructure.cluster.x-k8s.io" "controllerKind"="AzureCluster" "kind"="AzureCluster" "name"="t-zryehue" "namespace"="org-t-cisxbee" "reconcileID"="6584a069-c708-41a7-aeab-df9d7b8055d6" "x-ms-correlation-request-id"="80d34972-2b00-4764-8ba1-922280a9aed8"

https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/472d76774951f3d203aa9497b7a83fd1aad49548/controllers/azureidentity_controller.go#L133

seems like we do have some sort of race condition