FundingCircle / jackdaw

A Clojure library for the Apache Kafka distributed streaming platform.
https://fundingcircle.github.io/jackdaw/
BSD 3-Clause "New" or "Revised" License
369 stars 80 forks source link

Add divert? functions #238

Open bn-darindouglass-zz opened 4 years ago

bn-darindouglass-zz commented 4 years ago

Bringing in context from Slack:

Some background: for one of our kstreams apps we have a util function call divert? that'll push off a record if a given predicate is true (i.e. to a failure topic, etc):

(defn divert? [stream pred topic]
  (let [[divert-stream continue-stream] (j/branch stream [pred (constantly true])]
    (j/to divert-stream topic)
    continue-stream)

This abstraction, IMO, would feel really good in the library proper because it provides a useful layer on top of j/branch to handle what i'd think is a common use-case (again, failure topics, etc);

mt3593 commented 4 years ago

This is quite a common pattern, we opted for having a list of predicates as you tend to want to check multiple things before moving on, so [stream topic & preds] as the signature. Still yes this would be a helpful thing to add to jackdaw

bn-darindouglass-zz commented 4 years ago

@mt3593 this is how the fn looks now (taken from the tests):

(-> builder
    (k/kstream topic-a)
    (k/divert? {mod-four? topic-four
                mod-five? topic-five
                mod-seven? topic-seven})
    (k/to topic-rest)))