Closed jvstein closed 1 year ago
merlimat
commented
1 year ago @jvstein This looks the same as #84 which was fixed by #87 and released in the 3.1.0 release.
There's probably something in your setup that is still using the 3.0.0 release? I just did a quick test with the above code and it works in 3.1.0 and fails as expected in 3.0.0.
jvstein
commented
1 year ago Thanks for linking #84. I had read that previously but couldn't find it anymore.
I'm definitely using pulsar-client==3.1.0, which is why I thought #87 might not be a full solution.
$ pip freeze
apache-bookkeeper-client==4.16.1
certifi==2023.5.7
fastavro==0.24.0
grpcio==1.27.2
prometheus-client==0.16.0
protobuf==3.20.3
pulsar-client==3.1.0
pymmh3==0.0.5
pytz==2023.3
ratelimit==2.2.1
six==1.16.0I just spun up a local standalone pulsar instance in docker and I'm not able to reproduce against that instance. I can still reproduce against my Kubernetes deployment (3 zookeeper, 4 bookies, 3 brokers, 1 recovery, 3 proxy) based on the helm chart (which I now see is pulsar 2.10.2, not 3.0.0).
merlimat
commented
1 year ago Can you get a gdb stack trace when the process is deadlocked?
jvstein
commented
1 year ago Sure.
merlimat
commented
1 year ago Can you also get the bt all to see all 4 threads?
jvstein
commented
1 year ago Attached. For what it's worth, this is the stack from my real code that's deadlocking not the exact example above. The fact that I'm using an avro schema might be relevant as there would likely be some additional communication with the schema registry. The schema itself isn't that complicated (five fields, no significant nesting) and has been used before on the cluster.
schema_file = "/path/to/my.avsc"
schema_definition = load_schema(schema_file)
schema = AvroSchema(None, schema_definition=schema_definition)
client = pulsar.Client('pulsar://my-pulsar-host:6650')
producer = client.create_producer(
topic="persistent://tenant/namespace/topic",
schema=schema,
block_if_queue_full=True,
batching_enabled=True,
batching_max_publish_delay_ms=10
)Gotcha, it looks a deadlock between threads 1 & 2.
There's a place where the Python GIL is not released before making the Pulsar C++ producer.flushAsync().
Getting a fix out.
jvstein
commented
1 year ago Cool. I was finally able to reproduce with the example code above and a docker based standalone cluster, but I needed to do it with a partitioned topic with at least 2 partitions.
merlimat
commented
1 year ago @jvstein thanks for the debug info. I've posted a fix and push a test tag to build wheel files with it.
Once they're done, you can try with one of the wheels from: https://github.com/merlimat/pulsar-client-python/actions/runs/5073247045
Describe the bug
The example code in the python client's
send_asyncmethod does not work for processes that end.The example needs a
producer.flush()call, but adding it introduces a deadlock.To Reproduce
Run this modified version of the example code against a pulsar cluster. Note changes to reduce output text, change from
while Truetofor i in range(500000), and introduce theproducer.flush()call. Theflushcall is necessary to correctly produce all messages. If omitted, some messages will be skipped.Desktop (please complete the following information):
pulsar-client==3.1.0 Python 3.9.13 Ubuntu 22.04.2 LTS Pulsar 2.10.2 running in a Kubernetes cluster; client connected via pulsar-proxy running as a LoadBalancer Service
Additional context / Fix
Adding a
time.sleep(1)before theproducer.flush()call allows the process to exit cleanly every time that I've tested. This seems to suggest that the deadlock occurs somewhere between trying to clear the active batch and trying to flush the producer.Possibly related: https://github.com/apache/pulsar/issues/5666