Open technomage opened 5 years ago
@technomage which compression types are you using? The implementation of compression support so far has mostly been due to the needs of someone who has time and know-how to do the implementation :/
Confluent uses one not currently supported which crashes a client trying to read from those topics.
@technomage lz4 isn't yet supported, but a PR would be welcome:
https://cwiki.apache.org/confluence/display/KAFKA/Compression
lz4 is type 3
- it would be added to https://github.com/kafkaex/kafka_ex/blob/76c752b82c48d04281b8fcf4ad4f14c4fbed335b/lib/kafka_ex/compression.ex
to basically wrap the lz4 compress/decompress operations.
Currently kafka supports more compression types by default than kafka_ex which can cause a client to fail when attempting to read such a topic. In particular confluent platform uses some of the newer compression types. At a minimum it would be nice to be able to configure kafka_ex to not die on such topics.