Issue with kafka transport

[ACodingfreak]

Hi All,

I am using libbgpstream to process route-view live kafka stream

Broker: stream.routeviews.org:9092 Topic: routeviews.linx.6939.bmp_raw

After around 7+ hours it fails as shown below where it looks like broker connection is down. Is this expected behavior of routeviews brokers or there is a known issue as I am missing the configurations ?

%5|1677031859.110|REQTMOUT|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/0: Timed out HeartbeatRequest in flight (after 10803ms, timeout #0)
%4|1677031859.110|REQTMOUT|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
2023-02-21 18:10:59 36417: bs_transport_kafka.c:144: ERROR: Local: Timed out (-185): GroupCoordinator: 1 request(s) timed out: disconnect (after 30037413ms in state UP)
2023-02-21 18:11:09 36417: bs_transport_kafka.c:144: ERROR: Local: Host resolution failure (-193): GroupCoordinator: Failed to resolve 'stream.routeviews.org:9092': Temporary failure in name resolution (after 10012ms in state CONNECT)
%4|1677031880.132|COMMITFAIL|rdkafka#consumer-1| [thrd:main]: Offset commit (unassign) failed for 1/1 partition(s): Local: Host resolution failure: routeviews.linx.6939.bmp_raw[0]@2453347383
%5|1677031907.343|REQTMOUT|rdkafka#consumer-1| [thrd:stream.routeviews.org:9092/bootstrap]: stream.routeviews.org:9092/0: Timed out FetchRequest in flight (after 60344ms, timeout #0)
%4|1677031907.343|REQTMOUT|rdkafka#consumer-1| [thrd:stream.routeviews.org:9092/bootstrap]: stream.routeviews.org:9092/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
2023-02-21 18:11:47 36417: bs_transport_kafka.c:144: ERROR: Local: Timed out (-185): stream.routeviews.org:9092/0: 1 request(s) timed out: disconnect (after 30086656ms in state UP)
2023-02-21 18:11:47 36417: bs_transport_kafka.c:144: ERROR: Local: All broker connections are down (-187): 2/2 brokers are down
2023-02-21 18:11:47 36417: bs_transport_kafka.c:330: ERROR: Unhandled Kafka error: Local: Timed out: GroupCoordinator response error: Local: Timed out
2023-02-21 18:11:47 36417: bgpstream_parsebgp_common.c:551: ERROR: Could not refill buffer

[alistairking]

This feels like connectivity issues on your end. I have services that consume from this stream for weeks/months without issue.

[ACodingfreak]

@alistairking - Thanks for the reply.

Well, I was suspecting the same.

I am trying out again with the latest librdkafka library (v2.0.2) instead of the default one provided for Ubuntu20.04. Are there any probable logs which I can enable on librdkafka end or steps which I can follow to confirm the network latency ?