Parsely / pykafka

Apache Kafka client for Python; high-level & low-level consumer/producer, with great performance.
http://pykafka.readthedocs.org/
Apache License 2.0
1.12k stars 232 forks source link

Support librdkafka in the ManagedBalancedConsumer #778

Open waytai opened 6 years ago

waytai commented 6 years ago
Traceback (most recent call last):                                                                          
  File "pyconsume.py", line 30, in <module>                                                                 
    use_rdkafka = True                                                                                      
  File "/usr/local/lib/python2.7/dist-packages/pykafka/topic.py", line 214, in get_balanced_consumer        
    return cls(self, self._cluster, consumer_group, **kwargs)                                               
  File "/usr/local/lib/python2.7/dist-packages/pykafka/managedbalancedconsumer.py", line 188, in __init__   
    self.__class__.__name__))                                                                               
ImportError: use_rdkafka is not available for ManagedBalancedConsumer                                       
Exception AttributeError: "'ManagedBalancedConsumer' object has no attribute '_running'" in <bound method Ma
nagedBalancedConsumer.__del__ of <pykafka.managedbalancedconsumer.ManagedBalancedConsumer at 0x7f5a65a31b10 
(consumer_group=KAFKA_01)>> ignored
emmettbutler commented 6 years ago

At the time the ManagedBalancedConsumer was written, librdkafka didn't support Kafka-managed consumer groups, so this ImportError was added to make that clear. We should reexamine where librdkafka's support is these days, and fix this if possible. This will probably happen during Parse.ly's internal upgrade to latest Kafka later this year.

waytai commented 6 years ago

than you for your kindly reply

razor-1 commented 6 years ago

seems like https://github.com/edenhill/librdkafka/issues/601#issuecomment-208354621 indicates some of how this works, and the issue overall explains that librdkafka supports this now, if I'm reading it correctly