Intermittent Local_MsgTimedOut errors

[mindwerxkmg]

Description

When running for prolonged periods of time (days/weeks) our Kafka producer begins throwing a Local_MsgTimedOut error. Sometimes the issue goes away on its own. Other times we have to reboot the machine running the Kafka client in order to get it to start publishing and occasionally that does not work either.

How to reproduce

We cannot reproduce on purpose. Client connection issue occurs in stage and prod environments.

Checklist

Please provide the following information:

• [ ] Confluent.Kafka nuget version: 0.11.6
• [ ] Apache Kafka version: 0.10.0.2.0
• [ ] Client configuration: ClientConfig.txt
• [ ] Operating system: Happens with Windows Server 2008, Window Server 2012 R2. Have not had an occurrence on Windows 2016 (yet).
• [ ] Provide logs (with "debug" : "..." as necessary in configuration) PublisherLogTimeoutExample.txt
• [ ] Provide broker log excerpts
• [ ] Critical issue This is a critical issue!

[mhowlett]

this means librdkafka wasn't able to deliver the message according to the configured message.timeout.ms and message.send.max.retries values. This is unlikely to be caused by a problem with the client - it's probably broker or network related. I'd start by looking at the broker metrics for any problems with the cluster (assuming you have this set up - if not, start with that), or client logs (i think debug='broker' should be enough, but I might be forgetting something).

[anchitj]

@mindwerxkmg Is this still an issue? Can you provide debug logs?

[ksdvishnukumar]

@mhowlett and @mindwerxkmg may I know what is the fix for this?