search

kafkaex / kafka_ex

Kafka client library for Elixir
MIT License
598 stars 162 forks source link

Support for Streaming GenStage Producer #143

Closed zapient closed 7 years ago

zapient commented 8 years ago

http://elixir-lang.org/blog/2016/07/14/announcing-genstage/

Developers who maintain libraries that integrate with external data sources, be it a RabbitMQ, Redis or Apacha Kafka, can explore GenStage as an abstraction for consuming data from those sources. Library developers must implement producers and leave it up for their users to configure the consumer stages.

Firemango commented 8 years ago

As I read the Kafka protocol, it's not supported to specify max messages on a stream, so I guess the producer would have to keep track of the demand and how many messages are streaming through, and open/close the stream accordingly.

Does this make sense to anyone? If so, I could try to make a POC for this.

bjhaid commented 8 years ago

@Firemango thanks for volunteering go ahead and send in a PoC

dantswain commented 8 years ago

@Firemango We'll have to be very careful about how we do this. GenStage currently requires Elixir v1.3 and some folks using KafkaEx might not be prepared to upgrade. That shouldn't stop doing a PoC, we'll just have to think carefully about how to proceed if it works well.

Firemango commented 8 years ago

Alright, I will get started. @dantswain I will see how the PoC goes. Maybe it won't actually depend on any kafkaex internal code and could move it to an entirely separate repo, which can be merged in at a later time to reduce juggling versions.

bjhaid commented 7 years ago

I am closing this issue, we will not add GenStage support to kafka_ex, it expects count of messages while kafka only supports bytes to be consumed which does not necessarily translate to a certain number of messages. There are also several ways to implement several things with GenStage that makes it difficult to implement a generic implementation that would fit into a library