Open raghu-nandan-bs opened 1 year ago
@avthart @nickvth could you please try deploying this branch in your test environment and let me know if it fixes the issue (#580 )?
please add following under spec.template.spec.containers[0]
args:
- --enable-hash
- --log-level
- debug
@avthart @nickvth could you please try deploying this branch in your test environment and let me know if it fixes the issue (#580 )? please add following under
spec.template.spec.containers[0]
args: - --enable-hash - --log-level - debug
@raghu-nandan-bs Thanks!
Is there an image available in the registry for this branch which we can use?
@avthart @nickvth could you please try deploying this branch in your test environment and let me know if it fixes the issue (#580 )? please add following under
spec.template.spec.containers[0]
args: - --enable-hash - --log-level - debug
@raghu-nandan-bs Thanks!
Is there an image available in the registry for this branch which we can use?
Hi @avthart we can build an image using make image
command, are you running on arm or amd64? I can build one and push to my personal docker registry, I dont have access to spotahome's one.
@raghu-nandan-bs Would be great if you can provide anamd64
image. Let me know, so that we can test this change.
@avthart please try with raghunandanbs/redis-operator:ef0e26ed4444c2bb70f6c33456e73b4c1a5fee26
This looks good:
time="2023-04-13T18:06:11Z" level=debug msg="albert/rfs-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:71"
time="2023-04-13T18:06:11Z" level=debug msg="albert/rfs-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:79"
time="2023-04-13T18:06:11Z" level=debug msg="albert/rfr-s-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:131"
time="2023-04-13T18:06:11Z" level=debug msg="albert/rfr-readiness-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:141"
time="2023-04-13T18:06:11Z" level=debug msg="albert/rfr-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:117"
time="2023-04-13T18:06:11Z" level=debug msg="albert/rfr-test pdb is upto date, no need to apply changes..." service=k8s.podDisruptionBudget src="client.go:179"
time="2023-04-13T18:06:11Z" level=debug msg="albert/rfr-test statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
albert/rfr-test statefulset
is still updated every 30 seconds
Also see Kyverno logs:
I0413 18:18:41.385301 1 validation.go:123] webhooks/resource/validate "msg"="validation passed" "action"="validate" "gvk"={"group":"apps","version":"v1","kind":"StatefulSet"} "kind"="StatefulSet" "name"="rfr-test" "namespace"="albert" "operation"="UPDATE" "policy"="require-run-as-non-root-user" "resource"="albert/StatefulSet/rfr-test" "uid"="f5dde34e-a8a3-4478-a936-8f5f634896e3" "user"={"username":"system:serviceaccount:albert:redis-operator","uid":"8bbf0e0b-3347-4043-9274-76e29cd948ce","groups":["system:serviceaccounts","system:serviceaccounts:albert","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["redis-operator-67f65cb7c9-626j4"],"authentication.kubernetes.io/pod-uid":["551f5729-efed-4412-9d51-1c7ea72772f5"]}}
I0413 18:19:11.573949 1 validation.go:123] webhooks/resource/validate "msg"="validation passed" "action"="validate" "gvk"={"group":"apps","version":"v1","kind":"StatefulSet"} "kind"="StatefulSet" "name"="rfr-test" "namespace"="albert" "operation"="UPDATE" "policy"="require-run-as-non-root-user" "resource"="albert/StatefulSet/rfr-test" "uid"="b0bbf554-e5c7-44f7-b08a-c5ad62d9253a" "user"={"username":"system:serviceaccount:albert:redis-operator","uid":"8bbf0e0b-3347-4043-9274-76e29cd948ce","groups":["system:serviceaccounts","system:serviceaccounts:albert","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["redis-operator-67f65cb7c9-626j4"],"authentication.kubernetes.io/pod-uid":["551f5729-efed-4412-9d51-1c7ea72772f5"]}}
I0413 18:19:41.608323 1 validation.go:123] webhooks/resource/validate "msg"="validation passed" "action"="validate" "gvk"={"group":"apps","version":"v1","kind":"StatefulSet"} "kind"="StatefulSet" "name"="rfr-test" "namespace"="albert" "operation"="UPDATE" "policy"="require-run-as-non-root-user" "resource"="albert/StatefulSet/rfr-test" "uid"="d88bb0e8-d6d1-4778-959e-e065818c3620" "user"={"username":"system:serviceaccount:albert:redis-operator","uid":"8bbf0e0b-3347-4043-9274-76e29cd948ce","groups":["system:serviceaccounts","system:serviceaccounts:albert","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["redis-operator-67f65cb7c9-626j4"],"authentication.kubernetes.io/pod-uid":["551f5729-efed-4412-9d51-1c7ea72772f5"]}}
Created a new RedisFailover:
time="2023-04-13T18:09:29Z" level=debug msg="service created" namespace=albert service=k8s.service serviceName=rfs-test-20230413 src="service.go:87"
time="2023-04-13T18:09:29Z" level=debug msg="configMap created" configMap=rfs-test-20230413 namespace=albert service=k8s.configMap src="configmap.go:74"
time="2023-04-13T18:09:29Z" level=debug msg="configMap created" configMap=rfr-s-test-20230413 namespace=albert service=k8s.configMap src="configmap.go:74"
time="2023-04-13T18:09:29Z" level=debug msg="configMap created" configMap=rfr-readiness-test-20230413 namespace=albert service=k8s.configMap src="configmap.go:74"
time="2023-04-13T18:09:29Z" level=debug msg="configMap created" configMap=rfr-test-20230413 namespace=albert service=k8s.configMap src="configmap.go:74"
time="2023-04-13T18:09:29Z" level=debug msg="podDisruptionBudget created" namespace=albert podDisruptionBudget=rfr-test-20230413 service=k8s.podDisruptionBudget src="poddisruptionbudget.go:75"
time="2023-04-13T18:09:29Z" level=debug msg="statefulSet created" namespace=albert service=k8s.statefulSet src="statefulset.go:104" statefulSet=rfr-test-20230413
time="2023-04-13T18:09:29Z" level=debug msg="podDisruptionBudget created" namespace=albert podDisruptionBudget=rfs-test-20230413 service=k8s.podDisruptionBudget src="poddisruptionbudget.go:75"
time="2023-04-13T18:09:30Z" level=debug msg="deployment created" deployment=rfs-test-20230413 namespace=albert service=k8s.deployment src="deployment.go:99"
time="2023-04-13T18:09:30Z" level=debug msg="Number of redis mismatch, waiting for redis statefulset reconcile" namespace=albert redisfailover=test-20230413 src="handler.go:79"
time="2023-04-13T18:09:30Z" level=debug msg="object processed" controller-id=redisfailover object-key=albert/test-20230413 operator=redisfailover service=kooper.controller src="controller.go:279"
time="2023-04-13T18:09:41Z" level=debug msg="albert/rfs-test-20230413 service has a different resource hash, updating the object..." service=k8s.service src="client.go:71"
time="2023-04-13T18:09:41Z" level=debug msg="service updated" namespace=albert service=k8s.service serviceName=rfs-test-20230413 src="service.go:107"
time="2023-04-13T18:09:41Z" level=debug msg="albert/rfs-test-20230413 configmap has a different resource hash, updating the object..." service=k8s.configMap src="client.go:79"
time="2023-04-13T18:09:41Z" level=debug msg="configMap updated" configMap=rfs-test-20230413 namespace=albert service=k8s.configMap src="configmap.go:92"
time="2023-04-13T18:09:41Z" level=debug msg="albert/rfr-s-test-20230413 configmap has a different resource hash, updating the object..." service=k8s.configMap src="client.go:131"
time="2023-04-13T18:09:41Z" level=debug msg="configMap updated" configMap=rfr-s-test-20230413 namespace=albert service=k8s.configMap src="configmap.go:92"
time="2023-04-13T18:09:41Z" level=debug msg="albert/rfr-readiness-test-20230413 configmap has a different resource hash, updating the object..." service=k8s.configMap src="client.go:141"
time="2023-04-13T18:09:41Z" level=debug msg="configMap updated" configMap=rfr-readiness-test-20230413 namespace=albert service=k8s.configMap src="configmap.go:92"
time="2023-04-13T18:09:41Z" level=debug msg="albert/rfr-test-20230413 configmap has a different resource hash, updating the object..." service=k8s.configMap src="client.go:117"
time="2023-04-13T18:09:41Z" level=debug msg="configMap updated" configMap=rfr-test-20230413 namespace=albert service=k8s.configMap src="configmap.go:92"
time="2023-04-13T18:09:41Z" level=debug msg="albert/rfr-test-20230413 pdb has a different resource hash, updating the object..." service=k8s.podDisruptionBudget src="client.go:179"
time="2023-04-13T18:09:41Z" level=debug msg="podDisruptionBudget updated" namespace=albert podDisruptionBudget=rfr-test-20230413 service=k8s.podDisruptionBudget src="poddisruptionbudget.go:94"
time="2023-04-13T18:09:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:09:41Z" level=debug msg="statefulSet updated" namespace=albert service=k8s.statefulSet src="statefulset.go:185" statefulSet=rfr-test-20230413
time="2023-04-13T18:09:41Z" level=debug msg="albert/rfs-test-20230413 pdb has a different resource hash, updating the object..." service=k8s.podDisruptionBudget src="client.go:179"
time="2023-04-13T18:09:41Z" level=debug msg="podDisruptionBudget updated" namespace=albert podDisruptionBudget=rfs-test-20230413 service=k8s.podDisruptionBudget src="poddisruptionbudget.go:94"
time="2023-04-13T18:09:41Z" level=debug msg="albert/rfs-test-20230413 deployment has a different resource hash, updating the object..." service=k8s.deployment src="client.go:90"
time="2023-04-13T18:09:42Z" level=debug msg="deployment updated" deployment=rfs-test-20230413 namespace=albert service=k8s.deployment src="deployment.go:118"
Only the statefulset is updated every 30 seconds:
time="2023-04-13T18:15:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:15:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:16:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:16:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:17:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:17:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:18:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:18:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:19:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:19:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:20:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:20:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:21:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:21:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:22:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:22:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:23:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:23:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:24:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:24:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:25:11Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
time="2023-04-13T18:25:41Z" level=debug msg="albert/rfr-test-20230413 statefulset has a different resource hash, updating the object..." service=k8s.statefulSet src="client.go:102"
@raghu-nandan-bs
Any update on this one?
Hi @avthart sorry, I'll pick this up and close in few days.
That would be great!
Any update on this one?
@avthart could you please try running with image: raghunandanbs/redis-operator:700e8bd
in your test environment?
please add following under spec.template.spec.containers[0] for redis operator deployment
args:
- --enable-hash
- --log-level
- debug
I've updated the code and ran some basic tests. Once you can confirm that this works as expected, I'll add some test cases and finalize the PR.
Any update on this one?
@avthart could you please try running with image:
raghunandanbs/redis-operator:700e8bd
in your test environment?please add following under spec.template.spec.containers[0] for redis operator deployment
args: - --enable-hash - --log-level - debug
I've updated the code and ran some basic tests. Once you can confirm that this works as expected, I'll add some test cases and finalize the PR.
@avthart please let me know if you've tried the new image and if it works as expected..
Operator running with your version:
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
redis-operator 1/1 1 1 4m35s redis-operator raghunandanbs/redis-operator:700e8bd
Operators logs confirm that resources are up to date:
k logs redis-operator-66c5b64976-rhl69| grep 'no need to apply changes'
time="2023-10-11T06:01:23Z" level=debug msg="albert/rfr-test statefulset is upto date, no need to apply changes..." service=k8s.statefulSet src="client.go:108"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfs-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:73"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfs-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:81"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfrm-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:175"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfrs-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:184"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfr-s-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:137"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfr-readiness-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:147"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfr-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:123"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfr-test pdb is upto date, no need to apply changes..." service=k8s.podDisruptionBudget src="client.go:203"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfr-test statefulset is upto date, no need to apply changes..." service=k8s.statefulSet src="client.go:108"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfs-test pdb is upto date, no need to apply changes..." service=k8s.podDisruptionBudget src="client.go:203"
time="2023-10-11T06:01:53Z" level=debug msg="albert/rfs-test deployment is upto date, no need to apply changes..." service=k8s.deployment src="client.go:94"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfs-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:73"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfs-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:81"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfrm-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:175"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfrs-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:184"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfr-s-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:137"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfr-readiness-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:147"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfr-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:123"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfr-test pdb is upto date, no need to apply changes..." service=k8s.podDisruptionBudget src="client.go:203"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfr-test statefulset is upto date, no need to apply changes..." service=k8s.statefulSet src="client.go:108"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfs-test pdb is upto date, no need to apply changes..." service=k8s.podDisruptionBudget src="client.go:203"
time="2023-10-11T06:02:23Z" level=debug msg="albert/rfs-test deployment is upto date, no need to apply changes..." service=k8s.deployment src="client.go:94"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfs-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:73"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfs-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:81"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfrm-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:175"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfrs-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:184"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfr-s-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:137"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfr-readiness-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:147"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfr-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:123"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfr-test pdb is upto date, no need to apply changes..." service=k8s.podDisruptionBudget src="client.go:203"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfr-test statefulset is upto date, no need to apply changes..." service=k8s.statefulSet src="client.go:108"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfs-test pdb is upto date, no need to apply changes..." service=k8s.podDisruptionBudget src="client.go:203"
time="2023-10-11T06:02:53Z" level=debug msg="albert/rfs-test deployment is upto date, no need to apply changes..." service=k8s.deployment src="client.go:94"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfs-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:73"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfs-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:81"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfrm-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:175"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfrs-test service is upto date, no need to apply changes..." service=k8s.service src="client.go:184"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfr-s-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:137"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfr-readiness-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:147"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfr-test configmap is upto date, no need to apply changes..." service=k8s.configMap src="client.go:123"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfr-test pdb is upto date, no need to apply changes..." service=k8s.podDisruptionBudget src="client.go:203"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfr-test statefulset is upto date, no need to apply changes..." service=k8s.statefulSet src="client.go:108"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfs-test pdb is upto date, no need to apply changes..." service=k8s.podDisruptionBudget src="client.go:203"
time="2023-10-11T06:03:23Z" level=debug msg="albert/rfs-test deployment is upto date, no need to apply changes..." service=k8s.deployment src="client.go:94"
Checking version of the resources self:
k get deployments.apps rfs-test -o yaml | grep generation
generation: 2
Normally, this counter increases with every reconciliation of the operator. This looks good!
Questions:
Thanks!
@avthart thanks a lot for confirming, I'll proceed with writing test cases and submit the PR formally. I will target this by end of this week
Why is this a feature flag, and is it possible to enable it by default?
Since this is a new feature, would like to keep it in "beta" state and not break existing functionality. Any user who wants to use this optimization can opt-in with the flag. Once the feature is generally accepted, we can make it a default. @samof76 @dinesh-murugiah @ese please pitch in if there's any contrary opinion.
Can this fix be included in the upcoming 1.3.0 release?
I hope so, depends on the repo's owners :grimacing:
Hi @ese @samof76 could you please review?
Any update @raghu-nandan-bs @ese @samof76?
Fixes #580
Changes proposed on the PR:
CreateOrUpdate
, apply the changes only if there is a diff in hash.