search

python-zk / kazoo

Kazoo is a high-level Python library that makes it easier to use Apache Zookeeper.
https://kazoo.readthedocs.io
Apache License 2.0
1.3k stars 387 forks source link

"reconfig" always returns BadVersionError when "from_config" is set, even if it is set correctly #629

Open zbentley opened 4 years ago

zbentley commented 4 years ago

Summary: Kazoo appears to be supplying "from_config"/"config_version" incorrectly in the reconfig command.

Expected Behavior

  1. get('/zookeeper/config') yields a config with a version.
  2. reconfig(joining"some hosts", leaving=None, new_hosts=None, from_config=version), where version is the version from step 1, works correctly.
  3. If reconfig fails with BadVersionError, any subsequent get('/zookeeper/config') calls should return a different version.
  4. However, if I do echo 'reconfig -v THE_VERSION -add SOME_HOSTS | zkCli.sh, the reconfiguration succeeds, with the same version string as above.

Actual Behavior

  1. get('/zookeeper/config') yields a config with a version.
  2. reconfig(joining"some hosts", leaving=None, new_hosts=None, from_config=version), always returns BadVersionError whenever from_config is supplied. Only omitting from_config allows reconfiguration to succeed.
  3. After getting a BadVersionError, subsequent get('/zookeeper/config') return the same version as returned in step 1.

Additionally, the int type assertion for from_config seems incorrect, as config is a zxid type (documented here), meaning that values like 1000000d5 are valid.

Suspected Root Cause

I think that Kazoo is incorrectly serializing the from_config value as an int instead of a zxid, though I'm not quite sure how to serialize it correctly.

An erroneous type assertion is here, and erroneous serialization code is here.

However, removing the type assertion and changing the serialization code to write_string did not resolve the problem for me; I was able to supply a str to client.reconfig without failing the type assertion, but still received a BadVersionError.

Snippet to Reproduce the Problem


def parse_zk_config(raw):
    for line in raw.splitlines():
        line = line.strip()
        if line.startswith('version'):
            return line.split('=', 1)[-1]

c = KazooClient()
c.start()
version = parse_zk_config(c.get('/zookeeper/config')[0])
version = int(version)  # Fails unless ZK version happens to be an int-castable value
c.reconfig(joining='server.3=22.0.46.63:2182:2183:participant;2181', leaving=None, new_members=None, config_id=version)

That snippet always fails for me (BadVersionError on last line). No other reconfigurations occur with my test ensemble, so there's not a race (i.e. the BadVersionError). Subsequent calls to c.get('/zookeeper/config') confirm that the version isn't changing.

Furthermore, the following command succeeds (assuming an initial version value of 100000000):

echo 'reconfig -v 100000000 -add server.3=22.0.46.63:2182:2183:participant;2181' | zkCli.sh

Logs with logging in DEBUG mode

DEBUG:kazoo.client:Sending request(xid=1): GetData(path='/zookeeper/config', watcher=None)
DEBUG:kazoo.client:Received response(xid=1): ('server.1=22.0.29.211:2182:2183:participant;0.0.0.0:2181\nversion=100000000', ZnodeStat(czxid=0, mzxid=4294967296, ctime=0, mtime=1604673499180, version=-1, cversion=0, aversion=-1, ephemeralOwner=0, dataLength=73, numChildren=0, pzxid=0))
DEBUG:kazoo.client:Sending request(xid=2): Reconfig(joining='server.3=22.0.46.63:2182:2183:participant;2181', leaving=None, new_members=None, config_id='100000000')
DEBUG:kazoo.client:Received error(xid=2) BadVersionError((), {})

Specifications