search

redhat-cop / namespace-configuration-operator

The namespace-configuration-operator helps keeping configurations related to Users, Groups and Namespaces aligned with one of more policies specified as a CRs
Apache License 2.0
204 stars 55 forks source link

reconcile error triggered by some status fields to be set to invalid values #49

Closed bergerx closed 4 years ago

bergerx commented 4 years ago

Version 0.2.0 complains about setting some setting some status fields of the CR to invalid values on namespace creation, but keeps working. It logs the below error (human readable form here, raw details are below):

error:

NamespaceConfig.redhatcop.redhat.io "networkpolicy-isolate-namespaces" is invalid:
[
status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace: Invalid value: "null": 
status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace in body must be of type array: "null", 
status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces: Invalid value: "null": 
status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces in body must be of type array: "null"
]

stacktrace:

github.com/go-logr/zapr.(*zapLogger).Error
    /home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
    /home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    /home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
    /home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:88"

More details to replicate:

Operator is deployed with helm on a fresh minikube instance:

[minikube:namespace-configuration-operator] $ helm ls
NAME                                NAMESPACE                           REVISION    UPDATED                             STATUS      CHART                                   APP VERSION
namespace-configuration-operator    namespace-configuration-operator    1           2020-05-20 08:20:34.47425 +0100 IST deployed    namespace-configuration-operator-v0.2.0 v0.2.0     
[minikube:namespace-configuration-operator] $ helm get values namespace-configuration-operator
USER-SUPPLIED VALUES:
null
[minikube:namespace-configuration-operator] $

The namespaceconfig's status field is getting the updates:

[minikube:namespace-configuration-operator] $ kubectl get namespaceconfigs -o yaml
apiVersion: v1
items:
- apiVersion: redhatcop.redhat.io/v1alpha1
  kind: NamespaceConfig
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"redhatcop.redhat.io/v1alpha1","kind":"NamespaceConfig","metadata":{"annotations":{},"name":"networkpolicy-isolate-namespaces"},"spec":{"labelSelector":{"matchExpressions":[{"key":"team","operator":"Exists"}]},"templates":[{"objectTemplate":"apiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n  name: allow-from-same-namespace\n  namespace: {{ .Name }}\nspec:\n  podSelector:\n  ingress:\n  - from:\n    - podSelector: {}\n"},{"objectTemplate":"apiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n  name: allow-from-system-namespaces\n  namespace: {{ .Name }}\nspec:\n  podSelector:\n  ingress:\n  - from:\n    - namespaceSelector:\n        matchLabels:\n          type: system\n"}]}}
    creationTimestamp: "2020-05-20T07:23:35Z"
    finalizers:
    - namespace-config-operator
    generation: 5
    name: networkpolicy-isolate-namespaces
    resourceVersion: "1458301"
    selfLink: /apis/redhatcop.redhat.io/v1alpha1/namespaceconfigs/networkpolicy-isolate-namespaces
    uid: b588b4c9-3a3a-4b76-96ef-5ba5e4ae8baa
  spec:
    annotationSelector: {}
    labelSelector:
      matchExpressions:
      - key: team
        operator: Exists
    templates:
    - excludedPaths:
      - .spec.replicas
      - .metadata
      - .status
      objectTemplate: |
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          name: allow-from-same-namespace
          namespace: {{ .Name }}
        spec:
          podSelector:
          ingress:
          - from:
            - podSelector: {}
    - excludedPaths:
      - .metadata
      - .status
      - .spec.replicas
      objectTemplate: |
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        metadata:
          name: allow-from-system-namespaces
          namespace: {{ .Name }}
        spec:
          podSelector:
          ingress:
          - from:
            - namespaceSelector:
                matchLabels:
                  type: system
  status:
    conditions:
    - lastTransitionTime: "2020-05-20T15:56:48Z"
      message: Awaiting next reconciliation
      reason: Successful
      status: "True"
      type: ReconcileSuccess
    lockedResourceStatuses:
      networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace:
      - lastTransitionTime: "2020-05-20T15:56:47Z"
        message: Awaiting next reconciliation
        reason: Successful
        status: "True"
        type: ReconcileSuccess
      networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces:
      - lastTransitionTime: "2020-05-20T15:56:47Z"
        message: Awaiting next reconciliation
        reason: Successful
        status: "True"
        type: ReconcileSuccess
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I can see an error line from the operator when creating new namespace with a matching selector:

{"level":"info","ts":1589990207.0729768,"logger":"namespace-config-operator","msg":"Reconciling NamespaceConfig","Request.Namespace":"","Request.Name":"networkpolicy-isolate-namespaces"}
{"level":"info","ts":1589990207.4424787,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"controller_locked_object_networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace","source":"kind source: networking.k8s.io/v1, Kind=NetworkPolicy"}
{"level":"info","ts":1589990207.4426448,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"controller_locked_object_networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces","source":"kind source: networking.k8s.io/v1, Kind=NetworkPolicy"}
{"level":"error","ts":1589990207.450363,"logger":"lockedresourcecontroller","msg":"unable to update status for","object":{"kind":"NamespaceConfig","apiVersion":"redhatcop.redhat.io/v1alpha1","metadata":{"name":"networkpolicy-isolate-namespaces","selfLink":"/apis/redhatcop.redhat.io/v1alpha1/namespaceconfigs/networkpolicy-isolate-namespaces","uid":"b588b4c9-3a3a-4b76-96ef-5ba5e4ae8baa","resourceVersion":"1458273","generation":5,"creationTimestamp":"2020-05-20T07:23:35Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"redhatcop.redhat.io/v1alpha1\",\"kind\":\"NamespaceConfig\",\"metadata\":{\"annotations\":{},\"name\":\"networkpolicy-isolate-namespaces\"},\"spec\":{\"labelSelector\":{\"matchExpressions\":[{\"key\":\"team\",\"operator\":\"Exists\"}]},\"templates\":[{\"objectTemplate\":\"apiVersion: networking.k8s.io/v1\\nkind: NetworkPolicy\\nmetadata:\\n  name: allow-from-same-namespace\\n  namespace: {{ .Name }}\\nspec:\\n  podSelector:\\n  ingress:\\n  - from:\\n    - podSelector: {}\\n\"},{\"objectTemplate\":\"apiVersion: networking.k8s.io/v1\\nkind: NetworkPolicy\\nmetadata:\\n  name: allow-from-system-namespaces\\n  namespace: {{ .Name }}\\nspec:\\n  podSelector:\\n  ingress:\\n  - from:\\n    - namespaceSelector:\\n        matchLabels:\\n          type: system\\n\"}]}}\n"},"finalizers":["namespace-config-operator"]},"spec":{"labelSelector":{"matchExpressions":[{"key":"team","operator":"Exists"}]},"annotationSelector":{},"templates":[{"objectTemplate":"apiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n  name: allow-from-same-namespace\n  namespace: {{ .Name }}\nspec:\n  podSelector:\n  ingress:\n  - from:\n    - podSelector: {}\n","excludedPaths":[".spec.replicas",".metadata",".status"]},{"objectTemplate":"apiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n  name: allow-from-system-namespaces\n  namespace: {{ .Name }}\nspec:\n  podSelector:\n  ingress:\n  - from:\n    - namespaceSelector:\n        matchLabels:\n          type: system\n","excludedPaths":[".metadata",".status",".spec.replicas"]}]},"status":{"conditions":[{"type":"ReconcileSuccess","status":"True","reason":"Successful","message":"Awaiting next reconciliation","lastTransitionTime":"2020-05-20T15:56:47Z"}],"lockedResourceStatuses":{"networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace":null,"networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces":null}}},"error":"NamespaceConfig.redhatcop.redhat.io \"networkpolicy-isolate-namespaces\" is invalid: [status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace: Invalid value: \"null\": status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace in body must be of type array: \"null\", status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces: Invalid value: \"null\": status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces in body must be of type array: \"null\"]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*EnforcingReconciler).ManageSuccess\n\t/home/travis/gopath/pkg/mod/github.com/redhat-cop/operator-utils@v0.2.4/pkg/util/lockedresourcecontroller/enforcing-reconciler.go:136\ngithub.com/redhat-cop/namespace-configuration-operator/pkg/controller/namespaceconfig.(*ReconcileNamespaceConfig).Reconcile\n\t/home/travis/gopath/src/github.com/redhat-cop/namespace-configuration-operator/pkg/controller/namespaceconfig/namespaceconfig_controller.go:194\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1589990207.4504862,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"namespace-config-operator","request":"/networkpolicy-isolate-namespaces","error":"NamespaceConfig.redhatcop.redhat.io \"networkpolicy-isolate-namespaces\" is invalid: [status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace: Invalid value: \"null\": status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace in body must be of type array: \"null\", status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces: Invalid value: \"null\": status.lockedResourceStatuses.networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces in body must be of type array: \"null\"]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1589990207.544878,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"controller_locked_object_networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace","source":"channel source: 0xc0000c0cd0"}
{"level":"info","ts":1589990207.5450218,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"controller_locked_object_networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace"}
{"level":"info","ts":1589990207.5450394,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"controller_locked_object_networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace","worker count":1}
{"level":"info","ts":1589990207.5453825,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"controller_locked_object_networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces","source":"channel source: 0xc00088d770"}
{"level":"info","ts":1589990207.5455709,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"controller_locked_object_networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces"}
{"level":"info","ts":1589990207.5456991,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"controller_locked_object_networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces","worker count":1}
{"level":"info","ts":1589990207.5458038,"logger":"lockedresourcecontroller","msg":"reconcile called for","object":"networking.k8s.io/v1/NetworkPolicy/test1/allow-from-system-namespaces","request":"test1/allow-from-system-namespaces"}
{"level":"info","ts":1589990207.555765,"logger":"lockedresourcecontroller","msg":"reconcile called for","object":"networking.k8s.io/v1/NetworkPolicy/test1/allow-from-same-namespace","request":"test1/allow-from-same-namespace"}
{"level":"info","ts":1589990208.45162,"logger":"namespace-config-operator","msg":"Reconciling NamespaceConfig","Request.Namespace":"","Request.Name":"networkpolicy-isolate-namespaces"}
raffaelespazzoli commented 4 years ago

I have noticed this error too. Eventually the operator always converges to a stable state, but I agree that this error should be fixed.