Closed mendynew closed 9 years ago
Is there any traceback dumped in your python log file?
I didn't see any message in log file.
The implementation of ConsumerHandler is below:
class ConsumerHandler:
def __init__(self,consumerName,topic):
self.kafka = KafkaClient("127.0.0.1:9092")
self.topic = topic
self.consumer = SimpleConsumer(self.kafka, consumerName, self.topic,max_buffer_size=10000000)
def read(self):
msg = self.consumer.get_message()
if not msg:
return ''
return msg.message.value
def close(self):
self.consumer.commit()
self.kafka.close()
def count(self):
return self.consumer.pending()
It's my fault that I used time.sleep() function in next_tuple. Thanks a lot @poros
I'm new to storm and pyleus, and I encountered some problems when reading data from kafka.
The class ConsumerHandler is a wrapper for reading from kafka which is tested successfully. The complie is successful, but error happens when run in storm:
I don't know how to solve this problem, the only purpose I want to reach is just checking the topic consumer offset and unread messages in kafka queue. Thank you very much if anybody can help me.