FundingCircle / jackdaw

A Clojure library for the Apache Kafka distributed streaming platform.
https://fundingcircle.github.io/jackdaw/
BSD 3-Clause "New" or "Revised" License
369 stars 80 forks source link

Run consumer in thread throw error KafkaConsumer is not safe for multi-threaded access #350

Open khmelevskii opened 1 year ago

khmelevskii commented 1 year ago

I'm using poll-and-loop! function with infinity loop to process events

(defn ^:private poll-and-loop!
  [consumer processing-fn continue?]
  (let [poll-ms 5000]
    (loop []
      (when @continue?
        (let [records (jc/poll consumer poll-ms)]
          (when (seq records)
            (processing-fn records)
            (.commitSync consumer))
          (recur))))))

But I don't need to block main thread. When I try to run it in another thread, for example with future I have a problem with a few subscriber java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

How I should handle this case?