_ConsumeKafkaTopic never terminates.

mohaseeb / beam-nuggets

Collection of transforms for the Apache beam python SDK.

http://mohaseeb.com/beam-nuggets/

MIT License

87 stars 38 forks source link

_ConsumeKafkaTopic never terminates. #30

Open robertwb opened 4 years ago

robertwb commented 4 years ago

The _ConsumeKafkaTopic DoFn will read forever (unless the topic is closed), which will not allow the bundle to finish and elements to be checkpointed/processed downstream. This should be re-written as an SFD. (Note also that there's ongoing work to expose Java's KafkaIO in Python as a cross-laungage transform).

https://github.com/mohaseeb/beam-nuggets/blob/164903e739383671210ecdfd8d20e8e756435d86/beam_nuggets/io/kafkaio.py#L67

wzorgdrager commented 3 years ago

Any hints on how to implement this @robertwb ? The cross-language KafkaIO transform is not working for me either, see here. I have the exact problem as you described here when using this connector.