FundingCircle / jackdaw

A Clojure library for the Apache Kafka distributed streaming platform.
https://fundingcircle.github.io/jackdaw/
BSD 3-Clause "New" or "Revised" License
368 stars 80 forks source link

Add support for TopicNameExtractor in `jackdaw.streams/to` #314

Closed dysmento closed 1 year ago

dysmento commented 2 years ago

As of Kafka v2.0.0, it's been possible to dynamically select the topic to sink to on a per-message basis, using an interface called TopicNameExtractor. This enables some nice routing use cases. e.g., events marked urgent can go on the "urgent-priority" topic, and low priority events go to the "low-priority" topic, etc.

Confluent has a blog post from 2019 about this feature. See also KIP-303

Checklist

dakra commented 2 years ago

Nice. There is also this 3year old pr open that adds topicextractor #57

dysmento commented 2 years ago

I noticed #57 right after I submitted this! My approach follows the convention established by other interfaces, like Aggregator, KeyValueMapper, ValueJoiner, ProcessorSupplier, Transformer, etc. etc. You can see the pattern in https://github.com/FundingCircle/jackdaw/blob/master/src/jackdaw/streams/lambdas.clj