search

confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client
https://github.com/confluentinc/confluent-kafka-dotnet/wiki
Apache License 2.0
44 stars 857 forks source link

Consumer not consuming message after error #1552

Open tech7857 opened 3 years ago

tech7857 commented 3 years ago

Hi @mhowlett - we are seeing occasionally after some errors our consume stop consuming messages but the application is still running seems like it is in a zombie state. How to determine which errors can be ignored and which cannot be ignored. Below are few errors

Broker: coordinator load in progress

Disconnected(after 1234523726ms in up state)

also if we have 3 broker it is best to specify all the 3 broker in consumer config or even 1 works as i was seeing internally it connects to all 3?

please guide us.

thanks

filipeesch commented 3 years ago

Hi @tech7857 , try using KafkaFlow on top Confluent Client, it has some protections when unrecoverable errors occurs. Sometime the Consumer should be recreated when a fatal error occurs, for example, and KafkaFlow will handle it for you. Using KafkaFlow you can restart the Consumers in runtime if needed using the Admin Web API. Look here: https://github.com/farfetch/kafka-flow

tech7857 commented 3 years ago

Hi @mhowlett - any thoughts/suggestions.. we are already checking the isfatal flag but still app sometimes hit the zombie state in case of non fatal errors

Thanks