search

Spinoco / fs2-kafka

Kafka client for functional streams for scala (fs2)
MIT License
78 stars 14 forks source link

kafka 0.11.0 is not supported #6

Open er999mi opened 7 years ago

pchlupacek commented 7 years ago

We will look into that. However releases of kafka are backward compatible, so you may use the driver and talk to 0.11 ensemble like it would by 0.10.x ensemble.

cleverdeveloper commented 5 years ago

One really cool feature of the newest Kafka versions is ability to have the record headers (https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers).

This, for example, allows one to filter the messages quickly without the need to parse the body of the message.

pchlupacek commented 5 years ago

This is quite interesting feature. We are doing similar thing with lazy evaluation and memoization of that evaluation. In fact if you think about it, this is available if you use cats.Eval, and define own structure of messages.

From the document I assume, that you have to always read full message bytes, so adding headers will slightly hurt performance /though likely non-measurable way).

As for the implementation, I believe only scodec kafka protocols have to be changed to support this feature.