python-zk / kazoo

Kazoo is a high-level Python library that makes it easier to use Apache Zookeeper.
https://kazoo.readthedocs.io
Apache License 2.0
1.3k stars 387 forks source link

Kazoo hangs in multiprocessing #519

Open ZedMcChen opened 6 years ago

ZedMcChen commented 6 years ago

The following contrived multiprocessing code hangs when run. Kazoo seems to be waiting for its async_set to complete.

!/usr/bin/python3

from kazoo.client import KazooClient from multiprocessing import Pool

def init_zk_conn(zk_server): kz = KazooClient(zk_server, timeout=10) kz.start() return kz

def update_znode(kz, znode, msg=''): if not kz.exists(znode): kz.create(znode, value=msg.encode('utf-8')) else: kz.set(znode, msg.encode('utf-8'))

KZ = init_zk_conn('localhost:2181')

def update_node(node): znode = '/testkz/' + node update_znode(KZ, znode, 'updating')

if name == 'main': pool = Pool(2) node_list = ['one', 'two'] pool.map(update_node, node_list) print('Done')

ShireCiel commented 5 years ago

I found init Kazoo before fork, and reuse it after fork it will hang. It's looks similar. You may can try (do not init any kazoo client before using and )init their own kazoo in their own process

hungrybirder commented 4 years ago

Any progress on this issue?