search

redhat-cop / namespace-configuration-operator

The namespace-configuration-operator helps keeping configurations related to Users, Groups and Namespaces aligned with one of more policies specified as a CRs
Apache License 2.0
204 stars 55 forks source link

[BUG] Resources disappear after reconciliation #61

Closed saberkan closed 4 years ago

saberkan commented 4 years ago

Issue:

When installing version v0.2.1 resources still disappear after reconciliation. I though this was related to my environnement (when closing issue #55), but even with a different cluster and operator installation from scratch it still behaving the same. We tried to debug on OCP 4 and OCP 3 cluster, and the issue still remain.

How to reproduce

1 - Install v0.2.1 manually 2 - create and label project:

oc new-project saberkan
oc label namespace saberkan overcommit=limited

3 - Create CR oc apply -f examples/namespace-config/overcommit-limitrange.yaml 4 - Check CR created

oc describe limits -n saberkan
Name:       overcommit-limits
Namespace:  saberkan
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   cpu       -    -    -                -              100
Container   memory    -    -    -                -              1

5 - Edit examples/namespace-config/overcommit-limitrange.yaml and set memory: 2 6 - Apply edited file oc apply -f examples/namespace-config/overcommit-limitrange.yaml 7 - Check limit object:

while true; do  oc describe limits -n saberkan; oc get limits -n saberkan; done
...
Name:       overcommit-limits
Namespace:  saberkan
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   cpu       -    -    -                -              100
Container   memory    -    -    -                -              1
NAME                CREATED AT
overcommit-limits   2020-07-30T14:10:09Z
No resources found.
No resources found.
No resources found.
...

Additional logs

From OCP 4

{"level":"info","ts":1596118497.31407,"logger":"namespace-config-operator","msg":"Reconciling NamespaceConfig","Request.Namespace":"","Request.Name":"overcommit-limitrange"}
{"level":"info","ts":1596118497.3197956,"logger":"namespace-config-operator","msg":"Reconciling NamespaceConfig","Request.Namespace":"","Request.Name":"overcommit-limitrange"}
{"level":"info","ts":1596118497.3202727,"logger":"controller-runtime.controller","msg":"Stopping workers","controller":"controller_locked_object_v1/LimitRange/saberkan/overcommit-limits"}
I0730 14:14:58.559432       1 request.go:621] Throttling request took 1.047828077s, request: GET:https://172.30.0.1:443/apis/network.operator.openshift.io/v1?timeout=32s
{"level":"info","ts":1596118499.720901,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"controller_locked_object_v1/LimitRange/saberkan/overcommit-limits","source":"kind source: /v1, Kind=LimitRange"}
{"level":"info","ts":1596118499.8212698,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"controller_locked_object_v1/LimitRange/saberkan/overcommit-limits","source":"channel source: 0xc001fb1ae0"}
{"level":"info","ts":1596118499.8213296,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"controller_locked_object_v1/LimitRange/saberkan/overcommit-limits"}
{"level":"info","ts":1596118499.82134,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"controller_locked_object_v1/LimitRange/saberkan/overcommit-limits","worker count":1}

From OCP 3

{"level":"info","ts":1596116270.964597,"logger":"namespace-config-operator","msg":"Reconciling NamespaceConfig","Request.Namespace":"","Request.Name":"project"}

{"level":"error","ts":1596116270.9706802,"logger":"controller_patchlocker","msg":"unable to update status for","object":{"apiVersion":"redhatcop.redhat.io/v1alpha1","kind":"NamespaceConfig","name":"project"},"error":"Operation cannot be fulfilled on namespaceconfigs.redhatcop.redhat.io \"project\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/redhat-cop/operator-utils/pkg/util/lockedresourcecontroller.(*EnforcingReconciler).ManageSuccess\n\t/home/travis/gopath/pkg/mod/github.com/redhat-cop/operator-utils@v0.3.3/pkg/util/lockedresourcecontroller/enforcing-reconciler.go:170\ngithub.com/redhat-cop/namespace-configuration-operator/pkg/controller/namespaceconfig.(*ReconcileNamespaceConfig).Reconcile\n\t/home/travis/gopath/src/github.com/redhat-cop/namespace-configuration-operator/pkg/controller/namespaceconfig/namespaceconfig_controller.go:195\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90"}

{"level":"error","ts":1596116270.9708848,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"namespace-config-operator","request":"/project","error":"Operation cannot be fulfilled on namespaceconfigs.redhatcop.redhat.io \"project\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90"}

{"level":"info","ts":1596116271.971184,"logger":"namespace-config-operator","msg":"Reconciling NamespaceConfig","Request.Namespace":"","Request.Name":"project"}
raffaelespazzoli commented 4 years ago

@cnuland would you be able to look into this issue?

raffaelespazzoli commented 4 years ago

@saberkan I have been trying to reproduce it, but no success:

NAME                CREATED AT
overcommit-limits   2020-07-31T01:46:59Z
Name:       overcommit-limits
Namespace:  saberkan
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   cpu       -    -    -                -              100
Container   memory    -    -    -                -              2
NAME                CREATED AT
overcommit-limits   2020-07-31T01:46:59Z
Name:       overcommit-limits
Namespace:  saberkan
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   cpu       -    -    -                -              100
Container   memory    -    -    -                -              2
NAME                CREATED AT
overcommit-limits   2020-07-31T01:46:59Z
Name:       overcommit-limits
Namespace:  saberkan
Type        Resource  Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---  ---  ---------------  -------------  -----------------------
Container   cpu       -    -    -                -              100
Container   memory

not sure how to help, any suggestion?

raffaelespazzoli commented 4 years ago

I was actually able to reproduce it, but only with the operator deployed from OLM... Will look into it more tomorrow.

saberkan commented 4 years ago

Great, thank you @raffaelespazzoli, I couldn't neither reproduce it every single time. I though it was something with my environment. But it seems not, and the bug doesn't occur every single time but at least a majority of times in my case.

raffaelespazzoli commented 4 years ago

@saberkan I thin the last merge should fix the issue, but as you know it's not easy to reproduce. Would you be able to make a build from master and test it before I cut a release?

saberkan commented 4 years ago

@raffaelespazzoli thank you for the fix, I confirm It works now for me under. OCP 4.4.8, and OCP 3.11.200.