readiness failure

[mrafieee]

there is a readiness probe on the redis pods, which is failing for me

curl --silent -u $USERNAME:$PASSWORD -w '%{http_code}' --output /dev/null localhost:8080/v1/bootstrap | grep 200 && /opt/redislabs/bin/rladmin status nodes | grep node:$(cat /etc/opt/redislabs/node.id) | grep OK

returning Readiness probe failed: 200 cat: /etc/opt/redislabs/node.id: No such file or directory

[laurentdroin]

Hi Mohammad,

Can you open a Support Ticket (send an email to support@redislabs.com) ? We will likely need the logs that are generated by running the log_collector.py script from this repository. Finally, if you have changed the password for the admin user in the Redis Enterprise UI, that could be the issue (you would need to reset the password back to what it originally was and which is stored in the secret).

Laurent.

[JoshBargar]

@laurentdroin @amiramm What was the solution for this?

[laurentdroin]

Hi Josh,

The issue was that the REC object (representing the Redis Enterprise Cluster) was created too soon (at a time that the redis-enterprise-operator was not fully ready). This caused a race condition.

The error about the readiness probe (and the missing node.id) is very generic and doesn't tell us much besides the fact that the Redis Enterprise node that we are trying to bootstrap (i.e. that we are trying to create a new cluster from or that we are trying to add to an existing cluster) is not bootstrapped (this could be normal - at initialization time - or could be indicative of a deeper issue that needs to be investigated).

If you are experiencing the same symptoms, we would suggest you open a Support Ticket and attach the logs, as we suggested to Mohammad. We would review them to determine what happened.

Laurent.

[JoshBargar]

Hi Josh,

The issue was that the REC object (representing the Redis Enterprise Cluster) was created too soon (at a time that the redis-enterprise-operator was not fully ready). This caused a race condition.

The error about the readiness probe (and the missing node.id) is very generic and doesn't tell us much besides the fact that the Redis Enterprise node that we are trying to bootstrap (i.e. that we are trying to create a new cluster from or that we are trying to add to an existing cluster) is not bootstrapped (this could be normal - at initialization time - or could be indicative of a deeper issue that needs to be investigated).

If you are experiencing the same symptoms, we would suggest you open a Support Ticket and attach the logs, as we suggested to Mohammad. We would review them to determine what happened.

Laurent.

Will do, thanks Laurent.