zookeeper ruok command check occasionally fails

alfred-landrum commented 2 years ago

Description

The zookeeper liveness and readiness checks use the zookeeper ruok command like so:

OK=$(echo ruok | nc 127.0.0.1 $CLIENT_PORT)

# Check to see if zookeeper service answers
if [[ "$OK" == "imok" ]]; then
...
fi

However, if I just sit in a shell and try manually running that, eventually, I'll very occasionally get a blank response instead of the expected imok, even though the service seems fine, and running the command immediately again returns the expected string.

I believe the root issue is that we occasionally see a TCP reset from the zookeeper command socket:

From looking at the sequence numbers, I think it receives an ack from the client side after it's started shutting down the socket state. Even though the imok response is in flight, or already in the tcp receive buffers, nc doesn't read the socket data & write to its standard out.

I replaced nc with socat, and then ran a test on the zookeeper instance. I temporarily disabled the liveness and readiness checks, and with a tcpdump running, ran a script that tried to run the checks until failures. Using just nc would fail in a few seconds; using nc -q 0 (close write side as soon as possible) would fail in 10s of seconds. Using just plain socat (echo ruok | socat stdio tcp4:127.0.0.1:2181) would not fail, though via the tcpdump, I could verify that it had seen TCP reset packets.

We're updating our local zookeeper image to install and use socat instead of nc.

Importance

Under a loaded system, this has sometimes caused the liveness probe to fail with enough frequency to cause kubernetes to kill the pod, so I think its important that this be resolved.