search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
303 stars 3.15k forks source link

After the connection with the broker times out, there is no reconnection #4820

Closed intfish123 closed 3 months ago

intfish123 commented 3 months ago

Read the FAQ first: https://github.com/confluentinc/librdkafka/wiki/FAQ

Do NOT create issues for questions, use the discussion forum: https://github.com/confluentinc/librdkafka/discussions

Description

After the connection with the broker times out, there is no reconnection

logs: 

%5|1724150741.381|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 2903ms, timeout #0): possibly held back by preceeding ProduceRequest with timeout in 16419ms
%4|1724150741.381|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1724150741.381|FAIL|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): 1 request(s) timed out: disconnect (average rtt 1364.539ms) (after 368268ms in state UP)
%5|1724150753.612|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 3723ms, timeout #0): possibly held back by preceeding ProduceRequest with timeout in 52056ms
%4|1724150753.612|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1724150753.612|FAIL|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): 1 request(s) timed out: disconnect (average rtt 1395.641ms) (after 8006ms in state UP)
%5|1724150754.615|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 997ms, timeout #0)
%5|1724150754.615|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 951ms, timeout #1)
%5|1724150754.615|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 878ms, timeout #2)
%4|1724150754.615|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out 3 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1724150754.615|FAIL|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): 3 request(s) timed out: disconnect (average rtt 1394.078ms) (after 1001ms in state UP)
%5|1724150763.671|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 284ms, timeout #0): possibly held back by preceeding ProduceRequest with timeout in 59600ms
%5|1724150763.671|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 242ms, timeout #1)
%4|1724150763.671|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out 2 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1724150763.671|FAIL|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): 2 request(s) timed out: disconnect (average rtt 1403.278ms) (after 9007ms in state UP)
%5|1724150764.718|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 993ms, timeout #0)
%5|1724150764.718|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 992ms, timeout #1)
%5|1724150764.718|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 983ms, timeout #2)
%5|1724150764.718|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 901ms, timeout #3)
%4|1724150764.718|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out 4 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1724150764.718|FAIL|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): 4 request(s) timed out: disconnect (average rtt 1401.322ms) (after 1000ms in state UP)
%5|1724150765.721|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 994ms, timeout #0): possibly held back by preceeding ProduceRequest with timeout in 59003ms
%5|1724150765.721|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 993ms, timeout #1)
%5|1724150765.721|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 991ms, timeout #2)
%5|1724150765.721|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 950ms, timeout #3)
%5|1724150765.721|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 948ms, timeout #4)
%4|1724150765.722|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out 7 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1724150765.722|FAIL|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): 7 request(s) timed out: disconnect (average rtt 1399.336ms) (after 1001ms in state UP)
%5|1724150771.727|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 2449ms, timeout #0)
%5|1724150771.728|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 2094ms, timeout #1)
%5|1724150771.728|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 1576ms, timeout #2)
%5|1724150771.728|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 1322ms, timeout #3)
%5|1724150771.728|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 680ms, timeout #4)
%4|1724150771.728|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out 6 in-flight, 0 retry-queued, 1 out-queue, 1 partially-sent requests
%3|1724150771.728|FAIL|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): 7 request(s) timed out: disconnect (average rtt 1394.824ms) (after 6004ms in state UP)
%5|1724150772.730|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 993ms, timeout #0)
%5|1724150772.730|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 828ms, timeout #1)
%5|1724150772.730|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out ProduceRequest in flight (after 704ms, timeout #2)
%4|1724150772.730|REQTMOUT|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): Timed out 3 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1724150772.730|FAIL|rdkafka#producer-2| [thrd:[10.80.129.11:9092/1](http://10.80.129.11:9092/1)]: [10.80.129.11:9092/1](http://10.80.129.11:9092/1): 3 request(s) timed out: disconnect (average rtt 1392.931ms) (after 1000ms in state UP)

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

intfish123 commented 3 months ago

What's time the producer will not auto reconnect to broker? and will the heartbeat effect reconnecting?

intfish123 commented 3 months ago

It's my application logic error. I close it.