search

OT-CONTAINER-KIT / redis-operator

A golang based redis operator that will make/oversee Redis standalone/cluster/replication/sentinel mode setup on top of the Kubernetes.
https://ot-redis-operator.netlify.app/
Apache License 2.0
731 stars 207 forks source link

Unable to delete redis-sentinel when redis-replication was deleted first #935

Closed alita1991 closed 1 month ago

alita1991 commented 1 month ago

What version of redis operator are you using?

kubectl logs <_redis-operator_pod_name> -n <namespace>

redis-operator version: 0.16.0

Does this issue reproduce with the latest release? Yes

What operating system and processor architecture are you using (kubectl version)?

kubectl version Output
$ kubectl version
Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8-eks-adc7111

What did you do?

I deleted the redis-sentinel and redis-replication CRs at the same time via kubectl delete -f <sentinel_replication.yaml>

What did you expect to see? Redis-operator should be able to delete the redis-sentinel deployment

What did you see instead? The redis-sentinel CR cannot be deleted, a finalizer is in place.

Additional context The issue is pretty hard to reproduce, the workaround I found was to recreate the redis-replication

Logs

{"level":"error","ts":"2024-05-20T12:00:11Z","logger":"controller_redis","msg":"Redis statefulset update failed","Request.StatefulSet.Namespace":"integration","Request.StatefulSet.Name":"redis-multi-rep","error":"Operation cannot be fulfilled on statefulsets.apps \"redis-multi-rep\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.updateStatefulSet\n\t/workspace/k8sutils/statefulset.go:736\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.patchStatefulSet\n\t/workspace/k8sutils/statefulset.go:212\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateOrUpdateStateFul\n\t/workspace/k8sutils/statefulset.go:111\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateReplicationRedis\n\t/workspace/k8sutils/redis-replication.go:64\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisReplicationReconciler).Reconcile\n\t/workspace/controllers/redisreplication_controller.go:58\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-05-20T12:00:11Z","logger":"controller_redis","msg":"Cannot create replication statefulset for Redis","Request.StatefulSet.Namespace":"integration","Request.StatefulSet.Name":"redis-multi-rep","error":"Operation cannot be fulfilled on statefulsets.apps \"redis-multi-rep\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateReplicationRedis\n\t/workspace/k8sutils/redis-replication.go:76\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisReplicationReconciler).Reconcile\n\t/workspace/controllers/redisreplication_controller.go:58\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-05-20T12:00:11Z","msg":"Reconciler error","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","RedisReplication":{"name":"redis-multi-rep","namespace":"integration"},"namespace":"integration","name":"redis-multi-rep","reconcileID":"5fe948e8-3896-443c-a06d-0f92d4ea88a0","error":"Operation cannot be fulfilled on statefulsets.apps \"redis-multi-rep\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-05-20T12:00:11Z","logger":"controllers.RedisReplication","msg":"Failed to Get the role Info of the","redis pod":"redis-multi-rep-2","error":"dial tcp 100.64.16.16:6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.checkRedisServerRole\n\t/workspace/k8sutils/redis.go:552\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.GetRedisNodesByRole\n\t/workspace/k8sutils/redis.go:539\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisReplicationReconciler).Reconcile\n\t/workspace/controllers/redisreplication_controller.go:81\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-05-20T12:00:11Z","logger":"controllers.RedisReplication","msg":"Will reconcile redis operator in again 10 seconds","Request.Namespace":"integration","Request.Name":"redis-multi-rep"}
{"level":"error","ts":"2024-05-20T12:00:12Z","logger":"controllers.RedisSentinel","msg":"Failed to Get the role Info of the","redis pod":"redis-multi-rep-2","error":"dial tcp :6379: connect: connection refused","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.checkRedisServerRole\n\t/workspace/k8sutils/redis.go:552\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.GetRedisNodesByRole\n\t/workspace/k8sutils/redis.go:539\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.getRedisReplicationMasterIP\n\t/workspace/k8sutils/redis-sentinel.go:319\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.getSentinelEnvVariable\n\t/workspace/k8sutils/redis-sentinel.go:252\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.generateRedisSentinelContainerParams\n\t/workspace/k8sutils/redis-sentinel.go:151\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.RedisSentinelSTS.CreateRedisSentinelSetup\n\t/workspace/k8sutils/redis-sentinel.go:78\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.CreateRedisSentinel\n\t/workspace/k8sutils/redis-sentinel.go:53\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisSentinelReconciler).Reconcile\n\t/workspace/controllers/redissentinel_controller.go:65\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":"2024-05-20T12:37:30Z","msg":"Reconciler error","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","RedisReplication":{"name":"redis-multi-rep","namespace":"integration"},"namespace":"integration","name":"redis-multi-rep","reconcileID":"ecbe8309-8c51-4775-bfab-eb3e2ec4360a","error":"Operation cannot be fulfilled on redisreplications.redis.redis.opstreelabs.in \"redis-multi-rep\": StorageError: invalid object, Code: 4, Key: /registry/redis.redis.opstreelabs.in/redisreplications/integration/redis-multi-rep, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d14183da-0042-40a2-b974-ebadc9e95883, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-05-20T12:37:30Z","logger":"controllers.RedisReplication","msg":"Reconciling opstree redis replication controller","Request.Namespace":"integration","Request.Name":"redis-multi-rep"}
{"level":"error","ts":"2024-05-20T12:37:30Z","logger":"controllers.RedisSentinel","msg":"Failed to Execute Get Request","Request.Namespace":"integration","Request.Name":"redis-multi-sen","replication name":"redis-multi-rep","namespace":"integration","error":"redisreplications.redis.redis.opstreelabs.in \"redis-multi-rep\" not found","stacktrace":"github.com/OT-CONTAINER-KIT/redis-operator/k8sutils.getRedisReplicationMasterIP\n\t/workspace/k8sutils/redis-sentinel.go:300\ngithub.com/OT-CONTAINER-KIT/redis-operator/k8sutils.IsRedisReplicationReady\n\t/workspace/k8sutils/redis-replication.go:216\ngithub.com/OT-CONTAINER-KIT/redis-operator/controllers.(*RedisSentinelReconciler).Reconcile\n\t/workspace/controllers/redissentinel_controller.go:48\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}