search

haskell-works / hw-kafka-client

Kafka client for Haskell, including auto-rebalancing consumers
MIT License
139 stars 50 forks source link

Application hangs on exit when multiple consumers share the same group id #153

Closed torgeirsh closed 3 years ago

torgeirsh commented 3 years ago

I have an application that starts two threads, and each thread creates a consumer (with a bracket that closes it) that consumes from a separate topic. There's a sigterm handler that kills the threads, and console printouts show that the consumers get closed when sigterm is received. However, the application never returns to the OS, it just seems to hang after the main function is finished, as if waiting for a thread to complete. While debugging this, I noticed that if I give the consumers separate group ids, the application terminates without problems, it's just when they share the same group id that it seems to hang.

Is this expected/undefined behaviour because it's not allowed to use the same group id from within one application, or is there a bug somewhere?

Compiled with GHC 8.8.3, running in Docker with Ubuntu 20.04.

AlexeyRaga commented 3 years ago

I wonder if it can be due to https://github.com/haskell-works/hw-kafka-client/issues/151 Although I don't clearly see how it can be related to consumer groups...

It is indeed an unexpected behaviour, but I can't say from the top of my head what would trigger it... :-/ hw-kafka-client doesn't manage consumer groups directly, it is done by librdkafka...