KazooClient.stop() leaks worker threads if there are socket errors on start()

[josephaltmaier]

I recently had an issue with Kazoo leaking worker threads from the SequentialThreadingHandler when stopping the KazooClient after a start with errors.

The problem manifested as my application running out of memory. Looking through the logs I saw the following exceptions repeated many times:

{"time": "2017-02-26 00:34:00,536", "name": "pykafka.balancedconsumer", "level": "ERROR", "message": "Stopping consumer in response to error: [Errno 24] Too many open files"} Traceback (most recent call last): File "/path/to/my/project/venv/local/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 294, in start self._zookeeper_connection_timeout_ms) File "/path/to/my/project/venv/local/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 344, in _setup_zookeeper self._zookeeper.start() File "/path/to/my/project/venv/local/lib/python2.7/site-packages/kazoo/client.py", line 541, in start event = self.start_async() File "/path/to/my/project/venv/local/lib/python2.7/site-packages/kazoo/client.py", line 576, in start_async self._connection.start() File "/path/to/my/project/venv/local/lib/python2.7/site-packages/kazoo/protocol/connection.py", line 170, in start rw_sockets = self.handler.create_socket_pair() File "/path/to/my/project/venv/local/lib/python2.7/site-packages/kazoo/handlers/threading.py", line 165, in create_socket_pair return utils.create_socket_pair(socket) File "/path/to/my/project/venv/local/lib/python2.7/site-packages/kazoo/handlers/utils.py", line 142, in create_socket_pair client_sock, srv_sock = module.socketpair() error: [Errno 24] Too many open files {"time": "2017-02-26 00:34:00,537", "name": "MY_CLASS", "level": "ERROR", "message": "MY ERROR MESSAGE"} Traceback (most recent call last): File "/path/to/my/project/server/my_first_file.py", line 768, in run File "/path/to/my/project/server/my_first_file.py", line 181, in getEvent File "/path/to/my/project/server/my_first_file.py", line 149, in getNextMessage File "/path/to/my/project/server/my_first_file.py", line 54, in getNextMessage File "/path/to/my/project/server/my_second_file.py", line 126, in getNextMessage File "/path/to/my/project/server/my_third_file.py", line 481, in getNextMessage File "/path/to/my/project/venv/local/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 240, in __init__ self.start() File "/path/to/my/project/venv/local/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 302, in start self.stop() File "/path/to/my/project/venv/local/lib/python2.7/site-packages/pykafka/balancedconsumer.py", line 320, in stop self._zookeeper.stop() File "/path/to/my/project/venv/local/lib/python2.7/site-packages/kazoo/client.py", line 596, in stop self._connection._write_sock.send(b'\0') AttributeError: 'NoneType' object has no attribute 'send'

Every time I tried to start a KazooClient I was getting an exception on socket creation, trying to stop my KazooClient, getting another exception, and leaking both of the SequentialThreadingHandler's worker threads.

It looks like I ran out of file handles so when I called start my KazooClients successfully initialized the SequentialThreadingHandler and started the handler's workers but failed to create a socket pair and threw an exception. KazooClient.stop() calls self._connection._write_sock.send(b'\0') which will throw an AttribueError if self._connection._write_sock is None. The AttributeError will be thrown before stop calls _safe_close so the SequentialThreadingHandler's worker threads will never be stopped.

I have written a fix for this issue and will be submitting a PR shortly

[josephaltmaier]

PR to fix this issue: https://github.com/python-zk/kazoo/pull/418

[josephaltmaier]

Updated my comment to reflect the fact that the socketpair isn't used to talk to Zookeeper.

[bbangert]

Closed by #418