search

python-zk / kazoo

Kazoo is a high-level Python library that makes it easier to use Apache Zookeeper.
https://kazoo.readthedocs.io
Apache License 2.0
1.3k stars 386 forks source link

RuntimeError: Invalid error code #727

Closed nkormakov closed 7 months ago

nkormakov commented 1 year ago

Expected Behavior

No error, connection should be closed.

Actual Behavior

RuntimeError: Invalid error code

Snippet to Reproduce the Problem

zk = KazooClient(hosts = f'{ip}', verify_certs = False, timeout = 10)
zk.start()
zk.get_children(target_path)

It is happening when zookeeper running with a few extra jvm parameters

-Djava.security.auth.login.config=/etc/kafka/zookeeper_jaas.conf
-Dzookeeper.allowSaslFailedClients=false
-Dzookeeper.sessionRequireClientSASLAuth=true

Default library doesn't handle well error code -124 which declares that client should authorise himself. Our application is ok with closing connection altogether and not messing with it. I've fixed it with monkey patching:

def new_read_response(self, header, buffer, offset):
    client = self.client
    request, async_object, xid = client._pending.popleft()
    if header.zxid and header.zxid > 0:
        client.last_zxid = header.zxid
    if header.xid != xid:
        exc = RuntimeError('xids do not match, expected %r '
                           'received %r', xid, header.xid)
        async_object.set_exception(exc)
        raise exc

    # Determine if it's an exists request and a no node error
    exists_error = (header.err == NoNodeError.code and
                    request.type == Exists.type)

    # Set the exception if it's not an exists error
    if header.err and not exists_error:
        # -124 error handler
        if header.err == -124:
            async_object.set(False)
            async_object.set_exception(NoAuthError())
            return CLOSE_RESPONSE
        callback_exception = EXCEPTIONS[header.err]()
        self.logger.debug(
            'Received error(xid=%s) %r', xid, callback_exception)
        if async_object:
            async_object.set_exception(callback_exception)
    elif request and async_object:
        if exists_error:
            # It's a NoNodeError, which is fine for an exists
            # request
            async_object.set(None)
        else:
            try:
                response = request.deserialize(buffer, offset)
            except Exception as exc:
                self.logger.exception(
                    "Exception raised during deserialization "
                    "of request: %s", request)
                async_object.set_exception(exc)
                return
            self.logger.debug(
                'Received response(xid=%s): %r', xid, response)

            # We special case a Transaction as we have to unchroot things
            if request.type == Transaction.type:
                response = Transaction.unchroot(client, response)

            async_object.set(response)

        # Determine if watchers should be registered
        watcher = getattr(request, 'watcher', None)
        if not client._stopped.is_set() and watcher:
            if isinstance(request, (GetChildren, GetChildren2)):
                client._child_watchers[request.path].add(watcher)
            else:
                client._data_watchers[request.path].add(watcher)

    if isinstance(request, Close):
        self.logger.log(BLATHER, 'Read close response')
        return CLOSE_RESPONSE

kazoo.protocol.connection.ConnectionHandler._read_response = new_read_response

Logs with logging in DEBUG mode

[2023-08-08 13:59:25,212][INFO]: connection.py:639:_connect - Connecting to 192.xxx.xxx.xxx(192.xxx.xxx.xxx):2181, use_ssl: False
[2023-08-08 13:59:25,223][DEBUG]: connection.py:312:_submit - Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=10000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None)
[2023-08-08 13:59:25,236][INFO]: client.py:532:_session_callback - Zookeeper connection established, state: CONNECTED
[2023-08-08 13:59:25,236][DEBUG]: connection.py:312:_submit - Sending request(xid=1): GetChildren(path='/', watcher=None)
[2023-08-08 13:59:25,248][ERROR]: connection.py:631:_connect_attempt - Unhandled exception in connection loop
Traceback (most recent call last):
  File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 602, in _connect_attempt
    response = self._read_socket(read_timeout)
  File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 453, in _read_socket
    return self._read_response(header, buffer, offset)
  File "/opt/scanners/m_zookeeper.py", line 41, in new_read_response
    callback_exception = EXCEPTIONS[header.err]()
  File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/exceptions.py", line 54, in _invalid_error_code
    raise RuntimeError('Invalid error code')
RuntimeError: Invalid error code
[2023-08-08 13:59:25,248][INFO]: client.py:537:_session_callback - Zookeeper session closed, state: CLOSED
Exception in thread Thread-3 (zk_loop):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 512, in zk_loop
    if retry(self._connect_loop, retry) is STOP_CONNECTING:
  File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/retry.py", line 126, in __call__
    return func(*args, **kwargs)
  File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 552, in _connect_loop
    status = self._connect_attempt(host, hostip, port, retry)
  File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 602, in _connect_attempt
    response = self._read_socket(read_timeout)
  File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/protocol/connection.py", line 453, in _read_socket
    return self._read_response(header, buffer, offset)
  File "/opt/scanners/m_zookeeper.py", line 41, in new_read_response
    callback_exception = EXCEPTIONS[header.err]()
  File "/opt/scanners/venv/lib/python3.10/site-packages/kazoo/exceptions.py", line 54, in _invalid_error_code
    raise RuntimeError('Invalid error code')
RuntimeError: Invalid error code

Specifications

StephenSorriaux commented 1 year ago

Hello,

Thank you for the issue. Indeed the ZSESSIONCLOSEDREQUIRESASLAUTH error (-124) is not handled properly (reminds me of a PR where @ztzg said some errors were not handled at all in the lib).

Some documentation for reference: https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md?plain=1#L1615

nkormakov commented 1 year ago

Hi Stephen!

What is the proper way of handling it will be? I'm aware that my workaround is not up to standards yet, but it does not produces some secondary errors or halts

ceache commented 1 year ago

We need to add a new error definition, at the very least. This is easy.

What we were referring to, with @ztzg, was better handling of unknown errors, this is much more comprehensive work.

Your monkey patch seems fine to me as a workaround for now. What are you asking exactly?

On Wed, Aug 9, 2023, 07:58 nkormakov @.***> wrote:

Hi Stephen!

What is the proper way of handling it will be? I'm aware that my workaround is not up to standards yet, but it does not produces some secondary errors or halts

— Reply to this email directly, view it on GitHub https://github.com/python-zk/kazoo/issues/727#issuecomment-1671191318, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIFTHRJZXBKMN7E6R2B7ADXUN3N7ANCNFSM6AAAAAA3ISBRDM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

nkormakov commented 1 year ago

Ok, I'll look into that and make a PR for you to judge.

jeffwidman commented 7 months ago

Fixed by #743