search

OT-CONTAINER-KIT / redis-operator

A golang based redis operator that will make/oversee Redis standalone/cluster/replication/sentinel mode setup on top of the Kubernetes.
https://ot-redis-operator.netlify.app/
Apache License 2.0
731 stars 206 forks source link

Setting readOnlyRootFilesystem true makes redis-sentinel CrashLoopBackOff #970

Closed ifalex closed 1 week ago

ifalex commented 3 weeks ago

What version of redis operator are you using?

kubectl logs <_redis-operator_pod_name> -n <namespace>
{"level":"info","ts":"2024-06-05T19:58:27Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2024-06-05T19:58:27Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":"2024-06-05T19:58:27Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":"2024-06-05T19:58:27Z","msg":"starting server","kind":"health probe","addr":"[::]:8081"}
I0605 19:58:27.416182       1 leaderelection.go:250] attempting to acquire leader lease integration/6cab913b.redis.opstreelabs.in...
I0605 19:58:27.453271       1 leaderelection.go:260] successfully acquired lease integration/6cab913b.redis.opstreelabs.in
{"level":"info","ts":"2024-06-05T19:58:27Z","msg":"Starting EventSource","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis","source":"kind source: *v1beta2.Redis"}
{"level":"info","ts":"2024-06-05T19:58:27Z","msg":"Starting Controller","controller":"redis","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"Redis"}
{"level":"info","ts":"2024-06-05T19:58:27Z","msg":"Starting EventSource","controller":"redisreplication","controllerGroup":"redis.redis.opstreelabs.in","controllerKind":"RedisReplication","source":"kind source: *v1beta2.RedisReplication"}

redis-operator version: redis-operator version: redis-operator:0.16.0 (built image from tag)

Does this issue reproduce with the latest release? Yes

What operating system and processor architecture are you using (kubectl version)?

kubectl version Output
$ kubectl version
Client Version: v1.29.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.8+k3s1

What did you do?

What did you expect to see? As the file becomes readable only I expect here to be allowed to set an emptyDir from configuration as it can be performed for the other crds provided. Not allowing this results in bellow error CrashLoopBackOff

What did you see instead?

Running sentinel without TLS mode ACL_MODE is not true, skipping ACL file modification Starting sentinel service ..... 1:X 06 Jun 2024 06:26:21.394 # Sentinel config file /etc/redis/sentinel.conf is not writable: Read-only file system. Exiting... /usr/bin/entrypoint-sentinel.sh: line 38: /etc/redis/sentinel.conf: Read-only file system /usr/bin/entrypoint-sentinel.sh: line 18: /etc/redis/sentinel.conf: Read-only file system /usr/bin/entrypoint-sentinel.sh: line 76: /etc/redis/sentinel.conf: Read-only file system

ifalex commented 3 weeks ago

Looking through the structure of the service, observed that the issue happens due to entrypoint who tries to modify /etc/redis/sentinel.conf based on env variables that are obtained through Helm. A proposal to mitigate this issue would be to move the creation of sentinel.conf through k8s config management native mechanism which is configMap. That way the entrypoint script would not be needed anymore allowing the read-only file system capability avaiable.