search

kafkaex / kafka_ex

Kafka client library for Elixir
MIT License
596 stars 163 forks source link

Support current kafka compression types #361

Open technomage opened 5 years ago

technomage commented 5 years ago

Currently kafka supports more compression types by default than kafka_ex which can cause a client to fail when attempting to read such a topic. In particular confluent platform uses some of the newer compression types. At a minimum it would be nice to be able to configure kafka_ex to not die on such topics.

dantswain commented 4 years ago

@technomage which compression types are you using? The implementation of compression support so far has mostly been due to the needs of someone who has time and know-how to do the implementation :/

technomage commented 4 years ago

Confluent uses one not currently supported which crashes a client trying to read from those topics.

joshuawscott commented 4 years ago

@technomage lz4 isn't yet supported, but a PR would be welcome:

https://cwiki.apache.org/confluence/display/KAFKA/Compression lz4 is type 3 - it would be added to https://github.com/kafkaex/kafka_ex/blob/76c752b82c48d04281b8fcf4ad4f14c4fbed335b/lib/kafka_ex/compression.ex to basically wrap the lz4 compress/decompress operations.