Namespace operator pod is crashlooping after upgrade

[A749999]

Hi Team,

We have update the namespace configuration operator from 1.2.4 to 1.2.5 and observed the operator pod was keep getting crashlooped after upgrade and also getting the below error in manager container logs. please help us to resolve this issue.

2024-01-20T12:11:43Z INFO Observed a panic in reconciler: interface conversion: validation.Schema is validation.schemaValidation, not validation.NullSchema {"controller": "namespaceconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "NamespaceConfig", "NamespaceConfig": {"name":"ns-sa-ctl"}, "namespace": "", "name": "ns-sa-ctl", "reconcileID": "45df894e-e13d-4195-a0b7-f915c06bb869"} panic: interface conversion: validation.Schema is validation.schemaValidation, not validation.NullSchema [recovered] panic: interface conversion: validation.Schema is validation.schemaValidation, not validation.NullSchema

goroutine 186 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Reconcile.func1() /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:115 +0x1e5 panic({0x22be620?, 0xc0072c0fc0?}) /opt/hostedtoolcache/go/1.21.4/x64/src/runtime/panic.go:914 +0x21f github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(LockedResourceManager).validateLockedResources(0xc000724700, {0xc00072b000, 0x62, 0xc000457a60?}) /home/runner/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.3.7/pkg/util/lockedresourcecontroller/locked-resource-manager.go:341 +0x6f3 github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(LockedResourceManager).SetResources(0xc000724700, {0xc00072b000?, 0x62, 0x80}) /home/runner/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.3.7/pkg/util/lockedresourcecontroller/locked-resource-manager.go:82 +0x77 github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(LockedResourceManager).Restart(0xc000724700, {0x27c5e68, 0xc0007d8300}, {0xc00072b000, 0x62, 0x80}, {0x3677360, 0x0, 0x0}, 0x0, ...) /home/runner/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.3.7/pkg/util/lockedresourcecontroller/locked-resource-manager.go:222 +0x16c github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(EnforcingReconciler).UpdateLockedResourcesWithRestConfig(0xc00016afd0, {0x27c5e68, 0xc0007d8300}, {0x27db750?, 0xc0008d4000?}, {0xc00072b000, 0x62, 0x80}, {0x3677360, 0x0, ...}, ...) /home/runner/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.3.7/pkg/util/lockedresourcecontroller/enforcing-reconciler.go:117 +0x3bc github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(EnforcingReconciler).UpdateLockedResources(...) /home/runner/go/pkg/mod/github.com/redhat-cop/operator-utils@v1.3.7/pkg/util/lockedresourcecontroller/enforcing-reconciler.go:91 github.com/redhat-cop/namespace-configuration-operator/controllers.(NamespaceConfigReconciler).Reconcile(0xc00016afd0, {0x27c5e68, 0xc0007d8300}, {{{0x0, 0x0}, {0xc00057f0f0, 0x9}}}) /home/runner/work/namespace-configuration-operator/namespace-configuration-operator/controllers/namespaceconfig_controller.go:121 +0x83c sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Reconcile(0x27c5e68?, {0x27c5e68?, 0xc0007d8300?}, {{{0x0?, 0x21c61c0?}, {0xc00057f0f0?, 0x27b66d0?}}}) /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:118 +0xb7 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler(0xc00037d220, {0x27c5ea0, 0xc0001fcd20}, {0x233bc20?, 0xc0000655c0?}) /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:314 +0x365 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem(0xc00037d220, {0x27c5ea0, 0xc0001fcd20}) /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:265 +0x1c9 sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2.2() /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:226 +0x79 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2 in goroutine 73 /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.2/pkg/internal/controller/controller.go:222 +0x565

Thanks Siva M

[raffaelespazzoli]

Can you share the cr that is generating that issue. I expect in it to find a manifest of a type without a schema. Which u didn't think was possible anymore. Regardless the null pointer exception should be handled more gracefully.

On Sat, Jan 20, 2024, 7:22 AM A749999 @.***> wrote:

Hi Team,

We have update the namespace configuration operator from 1.2.4 to 1.2.5 and observed the operator pod was keep getting crashlooped after upgrade and also getting the below error in manager container logs. please help us to resolve this issue.

2024-01-20T12:11:43Z INFO Observed a panic in reconciler: interface conversion: validation.Schema is validation.schemaValidation, not validation.NullSchema {"controller": "namespaceconfig", "controllerGroup": "redhatcop.redhat.io", "controllerKind": "NamespaceConfig", "NamespaceConfig": {"name":"ns-sa-ctl"}, "namespace": "", "name": "ns-sa-ctl", "reconcileID": "45df894e-e13d-4195-a0b7-f915c06bb869"} panic: interface conversion: validation.Schema is validation.schemaValidation, not validation.NullSchema [recovered] panic: interface conversion: validation.Schema is validation.schemaValidation, not validation.NullSchema

goroutine 186 [running]:

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /home/runner/go/pkg/mod/ @.***/pkg/internal/controller/controller.go:115 +0x1e5 panic({0x22be620?, 0xc0072c0fc0?}) /opt/hostedtoolcache/go/1.21.4/x64/src/runtime/panic.go:914 +0x21f

github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*LockedResourceManager).validateLockedResources(0xc000724700, {0xc00072b000, 0x62, 0xc000457a60?}) /home/runner/go/pkg/mod/ @.***/pkg/util/lockedresourcecontroller/locked-resource-manager.go:341 +0x6f3

github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*LockedResourceManager).SetResources(0xc000724700, {0xc00072b000?, 0x62, 0x80}) /home/runner/go/pkg/mod/ @.***/pkg/util/lockedresourcecontroller/locked-resource-manager.go:82 +0x77

github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*LockedResourceManager).Restart(0xc000724700, {0x27c5e68, 0xc0007d8300}, {0xc00072b000, 0x62, 0x80}, {0x3677360, 0x0, 0x0}, 0x0, ...) /home/runner/go/pkg/mod/ @.***/pkg/util/lockedresourcecontroller/locked-resource-manager.go:222 +0x16c

github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*EnforcingReconciler).UpdateLockedResourcesWithRestConfig(0xc00016afd0, {0x27c5e68, 0xc0007d8300}, {0x27db750?, 0xc0008d4000?}, {0xc00072b000, 0x62, 0x80}, {0x3677360, 0x0, ...}, ...) /home/runner/go/pkg/mod/ @.***/pkg/util/lockedresourcecontroller/enforcing-reconciler.go:117 +0x3bc

github.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*EnforcingReconciler).UpdateLockedResources(.. .) /home/runner/go/pkg/mod/ @.***/pkg/util/lockedresourcecontroller/enforcing-reconciler.go:91

github.com/redhat-cop/namespace-configuration-operator/controllers.(*NamespaceConfigReconciler).Reconcile(0xc00016afd0, {0x27c5e68, 0xc0007d8300}, {{{0x0, 0x0}, {0xc00057f0f0, 0x9}}}) /home/runner/work/namespace-configuration-operator/namespace-configuration-operator/controllers/namespaceconfig_controller.go:121 +0x83c

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x27c5e68?, {0x27c5e68?, 0xc0007d8300?}, {{{0x0?, 0x21c61c0?}, {0xc00057f0f0?, 0x27b66d0?}}}) /home/runner/go/pkg/mod/ @.***/pkg/internal/controller/controller.go:118 +0xb7

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00037d220, {0x27c5ea0, 0xc0001fcd20}, {0x233bc20?, 0xc0000655c0?}) /home/runner/go/pkg/mod/ @.***/pkg/internal/controller/controller.go:314 +0x365

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00037d220, {0x27c5ea0, 0xc0001fcd20}) /home/runner/go/pkg/mod/ @.***/pkg/internal/controller/controller.go:265 +0x1c9

sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /home/runner/go/pkg/mod/ @./pkg/internal/controller/controller.go:226 +0x79 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).Start.func2 in goroutine 73 /home/runner/go/pkg/mod/ **@.***/pkg/internal/controller/controller.go:222 +0x565

Thanks Siva M

— Reply to this email directly, view it on GitHub https://github.com/redhat-cop/namespace-configuration-operator/issues/189, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPERXE4PTAALCLCXCSCU5DYPOZJRAVCNFSM6AAAAABCDE6Q4SVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA4TEMBTGQ3TANA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[A749999]

Thanks for your reply.. Please find the CR details which causing the issue.

apiVersion: redhatcop.redhat.io/v1alpha1 kind: NamespaceConfig metadata: creationTimestamp: "2021-09-28T11:02:48Z" finalizers:

• namespaceconfig-controller generation: 5 name: ns-sa-ctl resourceVersion: "1593732342" uid: 602f466e-fe17-425d-a85e-9b09cfb5bf6e spec: annotationSelector: {} labelSelector: matchLabels: nsctl: "true" templates:
• excludedPaths:
  • .metadata
  • .status
  • .spec.replicas objectTemplate: | apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: {{ .Name }}-sarole namespace: {{ .Name }} roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: sa-role subjects:
    • kind: ServiceAccount name: test-sa-namespace namespace: itt-dev
    • kind: ServiceAccount name: test-sa-namespace namespace: ocp-dev
• excludedPaths:
  • .metadata
  • .status
  • .spec.replicas objectTemplate: | apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: {{ .Name }}-authn-k8s-sa-oc-3-role namespace: {{ .Name }} roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:public-info-viewer subjects:
    • kind: ServiceAccount name: authn-k8s-sa-oc-3 namespace: conjur-follower-namespace status: conditions:
• lastTransitionTime: "2024-01-20T03:36:22Z" message: "" observedGeneration: 5 reason: LastReconcileCycleSucceded status: "True" type: ReconcileSuccess
• lastTransitionTime: "2023-09-14T10:54:51Z" message: 'Get "https://172.30.0.1:443/openapi/v2?timeout=32s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)' observedGeneration: 5 reason: LastReconcileCycleFailed status: "True" type: ReconcileError

[A749999]

Any update on this issue?

[raffaelespazzoli]

that issue should be solved in v1.2.6, can you try it?

[A749999]

that issue should be solved in v1.2.6, can you try it?

Great.. issue was resolved after we upgrade the operator to v1.2.6

[A749999]

issue was resolved after we upgrade the operator to v1.2.6