Is comsumer.consume() Idempotent?

[ChieveiT]

Can I call comsumer.consume() multiple times? Would there be any side effect such as multiple unexpected background threads for fetching?

[webmakersteve]

librdkafka (and the java kafka library) already have a concept of "fetch" vs "consume". Fetcher routines run (potentially in the background) to fetch data after the currently processed offset so future calls to consume may not require much network IO. librdkafka has dedicated threads for background processes, like fetch, so you are always fetching data provided you are keeping up with the stream on the partitions you have been assigned.

Idempotency is not guaranteed in consume because its nature is to not be idempotent. It keeps a cursor of where you are based on what you have consumed before, and gives you different data as you call the method multiple times.

Operations like seek or assign are idempotent, but consume by its nature cannot be since it is not fetching a particular message and is instead using a cursor.