search

spotahome / redis-operator

Redis Operator creates/configures/manages high availability redis with sentinel automatic failover atop Kubernetes.
Apache License 2.0
1.49k stars 357 forks source link

Redis and Sentinel pods fail liveness probes when authentication is enabled #575

Closed mpihlak closed 1 year ago

mpihlak commented 1 year ago

Expected behaviour

After enabling authentication on RedisFailover, expecting the pods to become ready and the cluster to become operational.

Actual behaviour

The pods are are created but never become ready:

 $ kubectl get pods -l app=auth-test-redis
NAME                                   READY   STATUS    RESTARTS   AGE
rfr-auth-test-redis-0                  1/2     Running   0          25m
rfr-auth-test-redis-1                  1/2     Running   0          25m
rfr-auth-test-redis-2                  1/2     Running   0          25m
rfs-auth-test-redis-598ffddc5f-dmqq2   1/2     Running   0          25m
rfs-auth-test-redis-598ffddc5f-l9mtm   1/2     Running   0          25m
rfs-auth-test-redis-598ffddc5f-qpxs9   1/2     Running   0          25m

The problem appears to be failing liveness probe:

  Warning  Unhealthy  4m40s (x65 over 14m)  kubelet            Readiness probe failed: unespected
AUTH failed: WRONGPASS invalid username-password pair or user is disabled.

The liveness probe is defined by the Redis Operator as:

sh -c redis-cli -h $(hostname) -p 6379 ping --user pinger --pass pingpass --no-auth-warning

Running this on one of the redis pods:

/data $ redis-cli -h localhost -p 6379 ping --user pinger --pass pingpass
(error) NOAUTH Authentication required.

However when changing the order of arguments so that the command comes last, it succeeds:

/data $ redis-cli -h localhost -p 6379 --user pinger --pass pingpass ping
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
PONG

So the thinking is that the command line arguments for the liveness check are passed in the wrong order by the operator. If this previously used to work then maybe the behavior has changed with later Redis version.

Steps to reproduce the behaviour

Create a secret:

kubectl create secret generic foo.redis --from-file=password

Add the auth clause to the redisfailover yaml file:

  auth:
    secretPath: foo.redis

kubectl apply the yaml.

Environment

mpihlak commented 1 year ago

Update. The issue with incorrect AUTH message was actually caused by having a newline at the end of the password. 

AUTH failed: WRONGPASS invalid username-password pair or user is disabled.

The above message actually originated from the readiness check, not the liveness. It became obvious that the extra newline was there after running sh -x /redis-readiness/ready.sh on one of the Redis nodes. After the fixing the secret the Redis pods started up and become ready.

The liveness probe still has the incorrect command line. However it does not fail because redis-cli incorrectly returns exit code 0 on authentication failure:

/data $ redis-cli -h localhost -p 6379 ping --user pinger --pass pingpass --no-auth-warning
(error) NOAUTH Authentication required.
/data $ echo $?
0

Suggest replacing that with something like:

sh -c "redis-cli -h localhost -p 6379 ping --user pinger --pass pingpass --no-auth-warning | grep PONG"