search

amaizfinance / redis-operator

Redis Operator for Kubernetes
Apache License 2.0
106 stars 31 forks source link

Deploying example not working #24

Closed iakat closed 4 years ago

iakat commented 4 years ago

Steps to reproduce: DigitalOcean Kubernetes v1.19.3

  1. Apply kustomization.yaml 
    bases:
- github.com/amaizfinance/redis-operator/deploy?ref=master
  2. Apply example/k8s_v1alpha1_redis_cr.yaml
  3. No master gets elected, no election happens

Output:

(⎈ |do-nyc3-automuteus:automuteus-redis-ha)~/g/automuteus-infra » k get pod,redis                                      master ✗
NAME                  READY   STATUS    RESTARTS   AGE
pod/redis-example-0   2/2     Running   0          5m2s
pod/redis-example-1   2/2     Running   0          4m43s
pod/redis-example-2   2/2     Running   0          4m27s

NAME                          MASTER   REPLICAS   DESIRED   AGE
redis.k8s.amaiz.com/example                       3         5m2s

Logs operator :

redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.409Z","logger":"controller_redis","msg":"Applied *v1.Service","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.422Z","logger":"controller_redis","msg":"Applied *v1.Service","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.433Z","logger":"controller_redis","msg":"Applied *v1.Service","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.446Z","logger":"controller_redis","msg":"Applied *v1.Service","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.452Z","logger":"controller_redis","msg":"Applied *v1.ConfigMap","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.463Z","logger":"controller_redis","msg":"Applied *v1beta1.PodDisruptionBudget","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.495Z","logger":"controller_redis","msg":"Applied *v1.StatefulSet","Namespace":"automuteus-redis-ha","Redis":"example"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.597Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.615Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.624Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.740Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:16.744Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:19.192Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 0 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:35.778Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 1 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:35.811Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 1 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:35.830Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 1 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"info","ts":"2020-11-10T23:04:39.674Z","logger":"controller_redis","msg":"Error creating Redis replication, requeue","Namespace":"automuteus-redis-ha","Redis":"example","error":"minimum replication size is not met, only 1 are healthy"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"error","ts":"2020-11-10T23:04:53.317Z","logger":"kubebuilder.controller","msg":"Reconciler error","controller":"redis-controller","request":"automuteus-redis-ha/example","error":"failed to update Pods: Pod \"redis-example-1\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1e\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-1\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-0\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-0\",\n  \t... // 15 identical fields\n  }\n;","stacktrace":"github.com/amaizfinance/redis-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tvendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"error","ts":"2020-11-10T23:04:54.371Z","logger":"kubebuilder.controller","msg":"Reconciler error","controller":"redis-controller","request":"automuteus-redis-ha/example","error":"failed to update Pods: Pod \"redis-example-2\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-2\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-1\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1e\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-1\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-0\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-0\",\n  \t... // 15 identical fields\n  }\n;","stacktrace":"github.com/amaizfinance/redis-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tvendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"error","ts":"2020-11-10T23:05:12.267Z","logger":"kubebuilder.controller","msg":"Reconciler error","controller":"redis-controller","request":"automuteus-redis-ha/example","error":"failed to update Pods: Pod \"redis-example-0\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-0\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-2\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-2\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-1\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1e\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-1\",\n  \t... // 15 identical fields\n  }\n;","stacktrace":"github.com/amaizfinance/redis-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tvendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
redis-operator-5675f8ccd-gszsm redis-operator {"level":"error","ts":"2020-11-10T23:05:13.297Z","logger":"kubebuilder.controller","msg":"Reconciler error","controller":"redis-controller","request":"automuteus-redis-ha/example","error":"failed to update Pods: Pod \"redis-example-2\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-2\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-0\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1g\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-0\",\n  \t... // 15 identical fields\n  }\n; Pod \"redis-example-1\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \t... // 10 identical fields\n  \tAutomountServiceAccountToken: nil,\n  \tNodeName:                     \"pool-kf1t8lj0w-3kj1e\",\n  \tSecurityContext: &core.PodSecurityContext{\n  \t\t... // 11 identical fields\n  \t\tFSGroupChangePolicy: nil,\n  \t\tSysctls:             nil,\n- \t\tSeccompProfile:      nil,\n+ \t\tSeccompProfile:      &core.SeccompProfile{Type: \"RuntimeDefault\"},\n  \t},\n  \tImagePullSecrets: nil,\n  \tHostname:         \"redis-example-1\",\n  \t... // 15 identical fields\n  }\n;","stacktrace":"github.com/amaizfinance/redis-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\tvendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/amaizfinance/redis-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\tvendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/amaizfinance/redis-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\tvendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

Redis logs

redis-example-1 redis 1:C 10 Nov 2020 23:04:40.264 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-example-1 redis 1:C 10 Nov 2020 23:04:40.264 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis-example-1 redis 1:C 10 Nov 2020 23:04:40.264 # Configuration loaded
redis-example-1 exporter time="2020-11-10T23:04:44Z" level=info msg="Redis Metrics Exporter v1.11.1    build date: 2020-08-28-17:21:19    sha1: 3d94cd439e70d3ab478bfa65c1f131ab978a60ad    Go: go1.15    GOOS: linux    GOARCH: amd64"
redis-example-1 exporter time="2020-11-10T23:04:44Z" level=info msg="Providing metrics at :9121/metrics"
redis-example-1 redis 1:M 10 Nov 2020 23:04:40.356 * Running mode=standalone, port=6379.
redis-example-1 redis 1:M 10 Nov 2020 23:04:40.357 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis-example-1 redis 1:M 10 Nov 2020 23:04:40.357 # Server initialized
redis-example-1 redis 1:M 10 Nov 2020 23:04:40.357 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
redis-example-1 redis 1:M 10 Nov 2020 23:04:40.358 * Ready to accept connections
redis-example-1 redis 1:S 10 Nov 2020 23:04:51.070 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
redis-example-1 redis 1:S 10 Nov 2020 23:04:51.071 * REPLICAOF 10.244.2.40:6379 enabled (user request from 'id=5 addr=10.244.2.63:56970 fd=8 name= age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=2 qbuf=122 qbuf-free=32646 argv-mem=4 obl=27 oll=0 omem=0 tot-mem=61468 events=r cmd=exec user=default')
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.165 * Connecting to MASTER 10.244.2.40:6379
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.166 * MASTER <-> REPLICA sync started
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.252 * Non blocking connect for SYNC fired the event.
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.255 * Master replied to PING, replication can continue...
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.259 * Trying a partial resynchronization (request 3c0c7132f4d0a76889d82a5867e8487cd36e56b3:1).
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.265 * Full resync from master: 6d8165b7d4cf8a33431579b0722238f74145b459:0
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.265 * Discarding previously cached master state.
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.322 * MASTER <-> REPLICA sync: receiving 175 bytes from master to disk
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.323 * MASTER <-> REPLICA sync: Flushing old data
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.323 * MASTER <-> REPLICA sync: Loading DB in memory
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.355 * Loading RDB produced by version 6.0.9
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.355 * RDB age 0 seconds
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.355 * RDB memory usage when created 1.85 Mb
redis-example-1 redis 1:S 10 Nov 2020 23:04:52.355 * MASTER <-> REPLICA sync: Finished with success
redis-example-2 redis 1:C 10 Nov 2020 23:04:52.739 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-example-2 redis 1:C 10 Nov 2020 23:04:52.739 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis-example-2 redis 1:C 10 Nov 2020 23:04:52.739 # Configuration loaded
redis-example-2 redis 1:M 10 Nov 2020 23:04:52.741 * Running mode=standalone, port=6379.
redis-example-2 redis 1:M 10 Nov 2020 23:04:52.827 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis-example-2 redis 1:M 10 Nov 2020 23:04:52.827 # Server initialized
redis-example-2 redis 1:M 10 Nov 2020 23:04:52.827 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
redis-example-2 redis-example-2 exporter time="2020-11-10T23:04:53Z" level=info msg="Redis Metrics Exporter v1.11.1    build date: 2020-08-28-17:21:19    sha1: 3d94cd439e70d3ab478bfa65c1f131ab978a60ad    Go: go1.15    GOOS: linux    GOARCH: amd64"
redis-example-2 exporter time="2020-11-10T23:04:53Z" level=info msg="Providing metrics at :9121/metrics"
redis 1:M 10 Nov 2020 23:04:52.828 * Ready to accept connections
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.138 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.138 * REPLICAOF 10.244.2.40:6379 enabled (user request from 'id=6 addr=10.244.2.63:44632 fd=8 name= age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=2 qbuf=122 qbuf-free=32646 argv-mem=4 obl=27 oll=0 omem=0 tot-mem=61468 events=r cmd=exec user=default')
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.961 * Connecting to MASTER 10.244.2.40:6379
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.962 * MASTER <-> REPLICA sync started
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.962 * Non blocking connect for SYNC fired the event.
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.962 * Master replied to PING, replication can continue...
redis-example-2 redis 1:S 10 Nov 2020 23:05:12.962 * Trying a partial resynchronization (request c0baa0b78d691bc078e06bf443220fbebe8fe262:1).
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.027 * Full resync from master: 6d8165b7d4cf8a33431579b0722238f74145b459:28
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.027 * Discarding previously cached master state.
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.048 * MASTER <-> REPLICA sync: receiving 175 bytes from master to disk
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.049 * MASTER <-> REPLICA sync: Flushing old data
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.049 * MASTER <-> REPLICA sync: Loading DB in memory
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.055 * Loading RDB produced by version 6.0.9
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.055 * RDB age 1 seconds
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.056 * RDB memory usage when created 1.88 Mb
redis-example-2 redis 1:S 10 Nov 2020 23:05:13.056 * MASTER <-> REPLICA sync: Finished with success
redis-example-0 redis 1:C 10 Nov 2020 23:04:18.631 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-example-0 redis 1:C 10 Nov 2020 23:04:18.631 # Redis version=6.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
redis-example-0 redis 1:C 10 Nov 2020 23:04:18.631 # Configuration loaded
redis-example-0 redis 1:M 10 Nov 2020 23:04:18.633 * Running mode=standalone, port=6379.
redis-example-0 redis 1:M 10 Nov 2020 23:04:18.633 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis-example-0 redis-example-0 exporter time="2020-11-10T23:04:19Z" level=info msg="Redis Metrics Exporter v1.11.1    build date: 2020-08-28-17:21:19    sha1: 3d94cd439e70d3ab478bfa65c1f131ab978a60ad    Go: go1.15    GOOS: linux    GOARCH: amd64"
redis 1:M 10 Nov 2020 23:04:18.633 # Server initialized
redis-example-0 redis 1:M 10 Nov 2020 23:04:18.633 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
redis-example-0 redis 1:M 10 Nov 2020 23:04:18.633 * Ready to accept connections
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.282 * Replica 10.244.2.181:6379 asks for synchronization
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.282 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '3c0c7132f4d0a76889d82a5867e8487cd36e56b3', my replication IDs are '599151193739e42bb7efd6e30649af575fa5fe60' and '0000000000000000000000000000000000000000')
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.282 * Replication backlog created, my new replication IDs are '6d8165b7d4cf8a33431579b0722238f74145b459' and '0000000000000000000000000000000000000000'
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.282 * Starting BGSAVE for SYNC with target: disk
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.283 * Background saving started by pid 36
redis-example-0 redis 36:C 10 Nov 2020 23:04:52.285 * DB saved on disk
redis-example-0 redis 36:C 10 Nov 2020 23:04:52.285 * RDB: 0 MB of memory used by copy-on-write
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.343 * Background saving terminated with success
redis-example-0 redis 1:M 10 Nov 2020 23:04:52.343 * Synchronization with replica 10.244.2.181:6379 succeeded
redis-example-0 redis 1:M 10 Nov 2020 23:05:12.962 * Replica 10.244.2.41:6379 asks for synchronization
redis-example-0 redis 1:M 10 Nov 2020 23:05:12.962 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'c0baa0b78d691bc078e06bf443220fbebe8fe262', my replication IDs are '6d8165b7d4cf8a33431579b0722238f74145b459' and '0000000000000000000000000000000000000000')
redis-example-0 redis 1:M 10 Nov 2020 23:05:12.962 * Starting BGSAVE for SYNC with target: disk
redis-example-0 redis 1:M 10 Nov 2020 23:05:12.963 * Background saving started by pid 62
redis-example-0 redis 62:C 10 Nov 2020 23:05:12.965 * DB saved on disk
redis-example-0 redis 62:C 10 Nov 2020 23:05:12.966 * RDB: 0 MB of memory used by copy-on-write
redis-example-0 redis 1:M 10 Nov 2020 23:05:13.048 * Background saving terminated with success
redis-example-0 redis 1:M 10 Nov 2020 23:05:13.048 * Synchronization with replica 10.244.2.41:6379 succeeded
redis-example-0 exporter time="2020-11-10T23:04:19Z" level=info msg="Providing metrics at :9121/metrics"
iakat commented 4 years ago

Cannot reproduce on k3s v1.18.9+k3s1. (seems to work as intended). Is 1.19 unsupported somehow?

iakat commented 4 years ago

Broken on 1.18.10-do.2.

iakat commented 4 years ago

Works on Scaleway Kapsule v1.19.3

nrvnrvn commented 4 years ago

@sim1 thanks for reporting that. I can confirm the issue with k3d and k8s v 1.19.3

clientVersion:
  buildDate: "2020-10-14T12:50:19Z"
  compiler: gc
  gitCommit: 1e11e4a2108024935ecfcb2912226cedeafd99df
  gitTreeState: clean
  gitVersion: v1.19.3
  goVersion: go1.15.2
  major: "1"
  minor: "19"
  platform: darwin/amd64
serverVersion:
  buildDate: "2020-11-13T07:19:02Z"
  compiler: gc
  gitCommit: 0e4fbfefe1dd8734756dfa4f9ab4fc89665cece4
  gitTreeState: clean
  gitVersion: v1.19.3+k3s3
  goVersion: go1.15.2
  major: "1"
  minor: "19"
  platform: linux/amd64

With this version operator fails to update pods' labels:

https://github.com/amaizfinance/redis-operator/blob/6baac7e8f80e3e668258f268e94d11afce565e92/pkg/controller/redis/redis_controller.go#L261-L305

Let me find another way to update labels without trying to update the whole Pod objects.

nrvnrvn commented 4 years ago

@sim1 please verify the fix for the latest image. k -n redis-operator set image deployments/redis-operator redis-operator=amaiz/redis-operator:latest

maeb commented 3 years ago

Image amaiz/redis-operator:latest works on kubernetes version v1.20.2.