search

mailgun / kafka-pixy

gRPC/REST proxy for Kafka
Apache License 2.0
768 stars 119 forks source link

Support for Kafka 0.11.0 #127

Closed hartfordfive closed 6 years ago

hartfordfive commented 6 years ago

Is Kafka 0.11.0 supported by this library or is this a pending upgrade?

horkhe commented 6 years ago

Kafka 0.11.0 is not yet supported. And this is not a library but a standalone application.

What particular feature of 0.11.0 are you interested in? Adding support to the extent of making Kafka-Pixy work with 0.11.0 is probably not a problem, but if you are looking for something like streaming and once-and-only-once guarantee then I do not have plans of implementing that.

hartfordfive commented 6 years ago

To be honest, it's not a particular feature in Kafka that I'm interested in. It's just a matter of using a more recent version. My use-case is nothing more than using kafka-pixy to publish data to multiple topics, so the streaming features and managing guarantees aren't required in this case.

horkhe commented 6 years ago

I see. So following these lines you probably also want support for v1.0.0. Kafka-Pixy uses Kafka protocol implementation from Shopify/sarama. Since that library has support for v0.11.0 in there latest version it should not be hard to add one to Kafka-Pixy. But v1.0.0 is not yet supported by Shopify/sarama so we need to wait for the library to catch up first.

Are you actually using Kafka-Pixy or just considering it?

hartfordfive commented 6 years ago

I am using it. To give you some context, it's being used in conjunction with an Openresty application which simply accepts incoming messages via HTTP and publishes them to Kafka. Unfortunately, there are no OpenResty compatible Lua libraries (stable and relatively mature) to publish to Kafka, so this is why i'm using kaka-pixy. There are other alternatives that could have been used, but without going into detail, this was the best option.

totemcaf commented 6 years ago

@horkhe you said that current version of Pixy is not going to work with Kafka 1.0.0 or that it will work but without new features available?

horkhe commented 6 years ago

@totemcaf I am working on making it work right now (ETA is the end of this week). But it will only work in the scope of the API Kafka-Pixy provides.