redhat-cop / namespace-configuration-operator

The namespace-configuration-operator helps keeping configurations related to Users, Groups and Namespaces aligned with one of more policies specified as a CRs
Apache License 2.0
204 stars 55 forks source link

can't delete resource - CR stuck #119

Closed davidkarlsen closed 9 months ago

davidkarlsen commented 3 years ago

I tried deleting a NamespaceConfiguration resource, but it won't be deleted due to a stuck finalizer, and the operator tries updating the object with a finalizer, which is not allowed because it is being deleted

2021-09-09T14:53:26.583Z        ERROR   controllers.NamespaceConfig     unable to update instance       {"namespaceconfig": "/fss-apps", "instance": {"apiVersion": "redhatcop.redhat.io/v1alpha1", "kind": "NamespaceConfig", "name": "fss-apps"}, "error": "NamespaceConfig.redhatcop.redhat.io \"fss-apps\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"namespaceconfig-controller\"}"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/redhat-cop/namespace-configuration-operator/controllers.(*NamespaceConfigReconciler).Reconcile
        /workspace/controllers/namespaceconfig_controller.go:85
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
raffaelespazzoli commented 3 years ago

can you paste the instance(yaml)? There should always ever be one finalizer. And the logic should be: if the instance is new, the finalizer is added, if the instance is being delete and the clean up has been completed the finalizer is removed...

davidkarlsen commented 3 years ago
--- a/manifests/governance/namespace-configurator/rolebindings.yaml
+++ /dev/null
@@ -1,44 +0,0 @@
-apiVersion: redhatcop.redhat.io/v1alpha1
-kind: NamespaceConfig
-metadata:
-  name: fss-apps
-spec:
-  labelSelector:
-    matchLabels:
-      app.kubernetes.io/managed-by: clout
-      fss.tietoevry.com/legacy-logging: "true"
-  templates:
-  - objectTemplate: |
-      apiVersion: rbac.authorization.k8s.io/v1
-      kind: RoleBinding
-      metadata:
-        name: fss-apps
-        namespace: {{ .Name }}
-        labels:
-          app.kubernetes.io/managed-by: namespace-configuration-operator
-      roleRef:
-        apiGroup: rbac.authorization.k8s.io
-        kind: ClusterRole
-        name: fss-apps
-      subjects:
-      - apiGroup: rbac.authorization.k8s.io
-        kind: Group
-        name: system:serviceaccounts:{{ .Name }}
raffaelespazzoli commented 3 years ago

sorry I meant the resource how it appears in the cluster with all of the fields.

On Fri, Sep 10, 2021 at 2:27 PM David J. M. Karlsen < @.***> wrote:

--- a/manifests/governance/namespace-configurator/rolebindings.yaml +++ /dev/null @@ -1,44 +0,0 @@ -apiVersion: redhatcop.redhat.io/v1alpha1 -kind http://redhatcop.redhat.io/v1alpha1-kind: NamespaceConfig -metadata:

  • name: fss-apps -spec:
  • labelSelector:
  • matchLabels:
  • app.kubernetes.io/managed-by: clout
  • fss.tietoevry.com/legacy-logging: "true"
  • templates:
    • objectTemplate: |
  • apiVersion: rbac.authorization.k8s.io/v1
  • kind: RoleBinding
  • metadata:
  • name: fss-apps
  • namespace: {{ .Name }}
  • labels:
  • app.kubernetes.io/managed-by: namespace-configuration-operator
  • roleRef:
  • apiGroup: rbac.authorization.k8s.io
  • kind: ClusterRole
  • name: fss-apps
  • subjects:
    • apiGroup: rbac.authorization.k8s.io
  • kind: Group
  • name: system:serviceaccounts:{{ .Name }}

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/redhat-cop/namespace-configuration-operator/issues/119#issuecomment-917119412, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPERXE2WIV7HAAXHJ34D6TUBJE25ANCNFSM5DXLARPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- ciao/bye Raffaele

davidkarlsen commented 3 years ago

sorry, that's gone by now. I'll reopen a case if I can reproduce.

davidkarlsen commented 2 years ago

@raffaelespazzoli found the cause, the finalizers seems to have changed name at some point, leading to having two:

finalizers:
  - namespace-config-operator
  - namespaceconfig-controller
raffaelespazzoli commented 2 years ago

this should be the right one: "namespaceconfig-controller". Also can you share between which version you see the change, for the benefit of others. And sorry for the inconvenience.

On Tue, Oct 19, 2021 at 2:54 PM David J. M. Karlsen < @.***> wrote:

@raffaelespazzoli https://github.com/raffaelespazzoli found the cause, the finalizers seems to have changed name at some point, leading to having two:

finalizers:

  • namespace-config-operator
  • namespaceconfig-controller

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/redhat-cop/namespace-configuration-operator/issues/119#issuecomment-947014976, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPERXADNOBC5DJICEHW2LTUHW5EVANCNFSM5DXLARPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- ciao/bye Raffaele

davidkarlsen commented 2 years ago

this should be the right one: "namespaceconfig-controller". Also can you share between which version you see the change, for the benefit of others. And sorry for the inconvenience. On Tue, Oct 19, 2021 at 2:54 PM David J. M. Karlsen < @.***> wrote: @raffaelespazzoli [@raffaelespazzoli](https://github.com/raffaelespazzoli) found the cause, the finalizers seems to have changed name at some point, leading to having two: finalizers: - namespace-config-operator - namespaceconfig-controller — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#119 (comment)>, or unsubscribe <github.com/notifications/unsubscribe-auth/ABPERXADNOBC5DJICEHW2LTUHW5EVANCNFSM5DXLARPQ> . Triage notifications on the go with GitHub Mobile for iOS <apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. -- ciao/bye Raffaele

I don't know in which version, I've upgraded several times.

davidkarlsen commented 2 years ago

Maybe a fix could be made in the controller to look for the invalid one and simply remove it?

raffaelespazzoli commented 2 years ago

one can simply create a little script that removes the old annotations.

On Mon, Oct 25, 2021 at 4:31 AM David J. M. Karlsen < @.***> wrote:

Maybe a fix could be made to look for the invalid one and simply remove it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/redhat-cop/namespace-configuration-operator/issues/119#issuecomment-950663748, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPERXHVJUK67HXAS7FWIXLUIUIW5ANCNFSM5DXLARPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- ciao/bye Raffaele