Quarkus Kafka: Retry attempts are not working as expected and application tries to connect to the Kafka Broker forever.

[krnkhanna]

Describe the bug I am working on a Quarkus application and intend to use Kafka to receive messages, however I want to stop the application if the application is not able to reach Kafka broker after retrying for a certain number of times. The default configuration is to try infinite number of times to reconnect. In the documentation at Smallrye Reactive Messaging Kafka, it says we can use kafka.retry-attempts or mp.messaging.incoming.[channel-name].retry-attempts to configure the number of retries. I have tried both but the application still goes on retring.

Expected behavior Application should stop retrying after a configured number of tries.

Actual behavior Application goes on retrying to connect to the broker.

To Reproduce Start the application without starting the Kafka broker or shut down Kafka broker after starting the application.

Steps to reproduce the behavior:

1. Try a small application which reads from a Kafka topic.
2. Start the application.
3. Shutdown the Kafka broker.
4. Application should stop retrying to connect after the configured number of times. This is not happening and the application keeps on trying.

Configuration

kafka.bootstrap.servers=localhost:9092
kafka.retry-attempts=2
mp.messaging.incoming.emea-auth-response-topic-incoming.retry-attempts=2

Screenshots (If applicable, add screenshots to help explain your problem.)

Environment (please complete the following information):

• Output of uname -a or ver:
• Output of java -version:
• GraalVM version (if different from Java):
• Quarkus version or git rev:
• Build tool (ie. output of mvnw --version or gradlew --version):

Additional context (Add any other context about the problem here.)

[jtama]

From the way the kafka-stream extension works, configuration entries picked must begin with quarkus. You should try :

quarkus.kafka.bootstrap.servers=localhost:9092
quarkus.kafka.retry-attempts=2

I think it would work bet if indeed you are using the kafka-streams extension.

[cescoffier]

I've reproduced the issue. It's the admin client (for health check) that don't get the right retry amount. In the meantime, disable the health check or just the readiness health check:

mp.messaging.incoming.emea-auth-response-topic-incoming.health-readiness-enabled=false

[cescoffier]

Actually, it won't be enough. The underlying Kafka consumer always attempt to reconnect.

[cescoffier]

We woulld need https://cwiki.apache.org/confluence/display/KAFKA/KIP-53+-+Add+custom+policies+for+reconnect+attempts+to+NetworkdClient which is not yet integrated.

[cescoffier]

With https://issues.apache.org/jira/browse/KAFKA-3496 marked as won't fix, there is not much we can do. Closing as it's an upstream client issue.

[ManishJain-TomTom]

@cescoffier , we are facing an issue opposite of this.

Once the connection breaks because of network issue or something, Quarkus application is not able to reconnect to the Kafka broker unless and until we explicitly restart the pod.

Are there any ways to fix that?