confluentinc / confluent-kafka-python

Confluent's Kafka Python Client
http://docs.confluent.io/current/clients/confluent-kafka-python
Other
84 stars 892 forks source link

Expose partitioners to python #1759

Open pavelschon opened 4 months ago

pavelschon commented 4 months ago

Description

We have use case where we need to manually calculate partition ID using the murmur2 partitioner from the librdkafka. Rationale: we need to verify that messages received belongs to the right partition.

Since the desired function is not exposed to python via confluent_kafka, we have a workaround with using ctypes, see below. But we still think the right approach is to use desired function (if available) via confluent_kafka module.

How to reproduce

librdkafka = ctypes.cdll.LoadLibrary("librdkafka.so.1")

assert librdkafka.rd_kafka_msg_partitioner_murmur2(None, key, len(key), num_partitions, None, None) == expected_partition

Checklist

Please provide the following information: