search

haskell-works / hw-kafka-client

Kafka client for Haskell, including auto-rebalancing consumers
MIT License
140 stars 50 forks source link

Add headers to consumed/produced records #179

Closed sigevsky closed 2 years ago

sigevsky commented 3 years ago

Closes #58

Implementation details

  1. I'm using [rd_kafka_produceva] (https://github.com/edenhill/librdkafka/blob/master/src/rdkafka.h#L4397) bindings (which is identical to rd_kafka_producev, besides structure passings instead of varargs) as regular rd_kafka_produce message doesn't support headers. This requires at least 1.5.0 librdkafka version: https://github.com/edenhill/librdkafka/issues/2895
  2. rd_kafka_produce_batch does not allow passing headers as well so I left ProduceRecord as is. Instead I've added a separate produceWithHeaders to cover the case when singular message is produced with headers.
  3. Consumer record now has crHeaders field which is a breaking change, but I don't really see how this can be done differently without duplicating the existing api.
sigevsky commented 3 years ago

@AlexeyRaga could you help me with pipeline/review?