We've been facing intermittent failures with the existing method of checking Zookeeper cluster readiness within our init container in the StatefulSet. The init container was echoing the mntr command to port 2181 of the Zookeeper service and grepping for zk_synced_followers. However, this check only works if the service directs the request to the leader of the Zookeeper pods. If a follower pod responds, the zk_synced_followers parameter is not present, causing the check to fail erroneously.
Solution
To address this issue, we're modifying the readiness check to echo the ruok command instead, and look for the imok response. This is the same method used in the Liveness and Readiness probes of the Zookeeper deployment.
This method of checking is more reliable and robust than the previous one, as it does not rely on the leader pod answering the readiness check and will work regardless of whether the service directs the request to the leader or a follower pod.
Testing
The problem with Stardog sporadically not reaching a ready state in Kubernetes has been reported by various customers, most recently IDA. I was easily able to reproduce the problem in my own k3s cluster as well as an Azure AKE cluster. This fix has been tested successfully in both k8s distributions with both 'Parallel' and 'OrderedReady' pod manangement policy.
Problem
We've been facing intermittent failures with the existing method of checking Zookeeper cluster readiness within our init container in the StatefulSet. The init container was echoing the
mntrcommand to port 2181 of the Zookeeper service and grepping forzk_synced_followers. However, this check only works if the service directs the request to the leader of the Zookeeper pods. If a follower pod responds, thezk_synced_followersparameter is not present, causing the check to fail erroneously.Solution
To address this issue, we're modifying the readiness check to echo the
ruokcommand instead, and look for theimokresponse. This is the same method used in the Liveness and Readiness probes of the Zookeeper deployment.This method of checking is more reliable and robust than the previous one, as it does not rely on the leader pod answering the readiness check and will work regardless of whether the service directs the request to the leader or a follower pod.
Testing
The problem with Stardog sporadically not reaching a ready state in Kubernetes has been reported by various customers, most recently IDA. I was easily able to reproduce the problem in my own k3s cluster as well as an Azure AKE cluster. This fix has been tested successfully in both k8s distributions with both 'Parallel' and 'OrderedReady' pod manangement policy.