search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
234 stars 3.15k forks source link

rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed - when the subscribed topic is deleted #2963

Closed dshivashankar1994 closed 4 years ago

dshivashankar1994 commented 4 years ago

I am using a cpp version of librdkafka consumer (version 1.4.2)

I am subscribe to 2 topics that are existing. They got assigned fine and was receiving messages. Now when I delete one of the topic, I was expecting 2 things (1) consumer.consume to throw the UNKNOWN_TOPIC_OR_PART error (2) Rebalance cb to revoke that partition

But both are not happening and instead I get the assertion error.

Logs in debug mode: Topics are team.topic.test.abc-1 and team.topic.test.abc-2. Deleted topic - team.topic.test.abc-1

[66536 (139930824054528) - DEBUG 20200703 20:49:47.969 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-1 [0]: stored offset 10, committed offset 10: not including in commit
[66536 (139930824054528) - DEBUG 20200703 20:49:47.969 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-2 [0]: stored offset 20, committed offset 20: not including in commit
[66536 (139930824054528) - DEBUG 20200703 20:49:53.245 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-1 [0]: stored offset 10, committed offset 10: not including in commit
[66536 (139930824054528) - DEBUG 20200703 20:49:53.245 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-2 [0]: stored offset 20, committed offset 20: not including in commit
[66536 (139930790483712) - DEBUG 20200703 20:49:55.302 IST KafkaReader.C#event_cb:184] 7, facility BROKERUA, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: team.topic.test.abc-1 [0]: broker unavailable: fetch: Broker: Unknown topic or partition
[66536 (139930790483712) - DEBUG 20200703 20:49:55.302 IST KafkaReader.C#event_cb:184] 7, facility FASTQUERY, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: Starting fast leader query
[66536 (139930790483712) - DEBUG 20200703 20:49:55.302 IST KafkaReader.C#event_cb:184] 7, facility FETCHADD, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Removed team.topic.test.abc-1 [0] from fetch list (0 entries, opv 2): fetch backed off
[66536 (139930824054528) - DEBUG 20200703 20:49:55.746 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
[66536 (139930824054528) - DEBUG 20200703 20:49:55.746 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Request metadata for 1 topic(s): partition leader query
[66536 (139930790483712) - DEBUG 20200703 20:49:55.803 IST KafkaReader.C#event_cb:184] 7, facility FETCHADD, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Added team.topic.test.abc-1 [0] to fetch list (1 entries, opv 2, 0 messages queued): fetchable
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: ===== Received metadata (for 1 requested topics): partition leader query =====
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #1/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #2/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility STATE, str [thrd:main]: Topic team.topic.test.abc-1 changed state exists -> notexists
[66536 (139930824054528) - INFO 20200703 20:49:55.951 IST KafkaReader.C#event_cb:204] 5, facility PARTCNT, str [thrd:main]: Topic team.topic.test.abc-1 partition count changed from 1 to 0
[66536 (139930824054528) - DEBUG 20200703 20:49:55.951 IST KafkaReader.C#event_cb:184] 7, facility REMOVE, str [thrd:main]: team.topic.test.abc-1 [0] no longer reported in metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility DESIRED, str [thrd:main]: Topic team.topic.test.abc-1 [0] is desired but no longer known: moving back on desired list
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BRKDELGT, str [thrd:main]: team.topic.test.abc-1 [0]: delegate to broker (none) (rktp 0x7f44080a8150, term 0, ref 7)
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BRKDELGT, str [thrd:main]: team.topic.test.abc-1 [0]: no longer delegated to broker sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BRKDELGT, str [thrd:main]: team.topic.test.abc-1 [0]: delegating to broker :0/internal for partition with 0 messages (0 bytes) queued
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BRKMIGR, str [thrd:main]: Migrating topic team.topic.test.abc-1 [0] 0x7f44080a8150 from sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3 to :0/internal (sending PARTITION_LEAVE to sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3)
[66536 (139930824054528) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: 1/1 requested topic(s) seen in metadata
[66536 (139930790483712) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility FETCHADD, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Removed team.topic.test.abc-1 [0] from fetch list (0 entries, opv 2): forced removal
[66536 (139930790483712) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility TOPBRK, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Topic team.topic.test.abc-1 [0]: leaving broker (0 messages in xmitq, next broker :0/internal, rktp 0x7f44080a8150)
[66536 (139930790483712) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BROADCAST, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: Broadcasting state change
[66536 (139930815661824) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility TOPBRK, str [thrd::0/internal]: :0/internal: Topic team.topic.test.abc-1 [0]: joining broker (rktp 0x7f44080a8150, 0 message(s) queued)
[66536 (139930815661824) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility BROADCAST, str [thrd::0/internal]: Broadcasting state change
[66536 (139930815661824) - DEBUG 20200703 20:49:55.952 IST KafkaReader.C#event_cb:184] 7, facility FETCHADD, str [thrd::0/internal]: :0/internal: Added team.topic.test.abc-1 [0] to fetch list (1 entries, opv 2, 0 messages queued): fetchable
[66536 (139930790483712) - DEBUG 20200703 20:49:56.155 IST KafkaReader.C#event_cb:184] 7, facility UNKTOPIC, str [thrd:sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/bootstra]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Received Fetch response (error 3) for unknown topic team.topic.test.abc-1 [0]: ignoring
[66536 (139930824054528) - DEBUG 20200703 20:49:56.245 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:56.245 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:56.245 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #1/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #2/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:56.451 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr.firm.com:9092/3: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:57.245 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:57.245 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:57.245 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:57.456 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:58.245 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-1 [0]: stored offset 10, committed offset 10: not including in commit
[66536 (139930824054528) - DEBUG 20200703 20:49:58.245 IST KafkaReader.C#event_cb:184] 7, facility OFFSET, str [thrd:main]: Topic team.topic.test.abc-2 [0]: stored offset 20, committed offset 20: not including in commit
[66536 (139930824054528) - DEBUG 20200703 20:49:58.246 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:58.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:58.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:58.472 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:59.246 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:49:59.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:59.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:49:59.488 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:50:00.246 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:50:00.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:50:00.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:50:00.505 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:50:01.246 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:50:01.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:50:01.246 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:50:01.526 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
...
...
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:50.013 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:50.808 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:50.808 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:50.808 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:51.029 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:51.030 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:51.808 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:51.808 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:51.809 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:52.045 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:52.809 IST KafkaReader.C#event_cb:184] 7, facility NOINFO, str [thrd:main]: Topic team.topic.test.abc-1 partition count is zero: should refresh metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:52.809 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:52.809 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 1 topic(s): refresh unavailable topics
[66536 (139930824054528) - DEBUG 20200703 20:52:52.967 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Requesting metadata for 2/2 topics: periodic topic and broker list refresh
[66536 (139930824054528) - DEBUG 20200703 20:52:52.967 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: Request metadata for 2 topic(s): periodic topic and broker list refresh
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 1 topics
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/1: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:53.063 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 1/1 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ===== Received metadata (for 2 requested topics): periodic topic and broker list refresh =====
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 4
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 5 brokers, 2 topics
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #0/5: mwkafka-staging-01.tbd.firm.com:9092 NodeId 5
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #1/5: mwkafka-staging-01.nyc.firm.com:9092 NodeId 1
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #2/5: mwkafka-staging-02.dr.firm.com:9092 NodeId 4
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #3/5: mwkafka-staging-02.nyc.firm.com:9092 NodeId 2
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Broker #4/5: mwkafka-staging-01.dr.firm.com:9092 NodeId 3
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #0/2: team.topic.test.abc-2 with 1 partitions
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]:   Topic team.topic.test.abc-2 partition 0 Leader 1
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1:   Topic #1/2: team.topic.test.abc-1 with 0 partitions: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: Error in metadata reply for topic team.topic.test.abc-1 (PartCnt 0): Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility METADATA, str [thrd:main]: sasl_plaintext://mwkafka-staging-01.nyc.firm.com:9092/1: 2/2 requested topic(s) seen in metadata
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility TOPICERR, str [thrd:main]: Subscribed topic not available: team.topic.test.abc-1: Broker: Unknown topic or partition
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility SUBSCRIPTION, str [thrd:main]: Group "team.subscriber.cpp.topic.jsh-abc": effective subscription list changed from 2 to 1 topic(s):
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility SUBSCRIPTION, str [thrd:main]:  Topic team.topic.test.abc-2 with 1 partition(s)
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility REJOIN, str [thrd:main]: Group "team.subscriber.cpp.topic.jsh-abc": subscription updated from metadata change: rejoining group
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility PAUSE, str [thrd:main]: Library pausing 2 partition(s)
console_team_subscriber: rd.h:326: rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed.
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility BARRIER, str [thrd:main]: team.topic.test.abc-1 [0]: rd_kafka_toppar_op_pause_resume:2383: new version barrier v3
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility PAUSE, str [thrd:main]: Pause team.topic.test.abc-1 [0] (v3)
[66536 (139930824054528) - DEBUG 20200703 20:52:53.368 IST KafkaReader.C#event_cb:184] 7, facility BARRIER, str [thrd:main]: team.topic.test.abc-2 [0]: rd_kafka_toppar_op_pause_resume:2383: new version barrier v3
Aborted (core dumped)
dshivashankar1994 commented 4 years ago

0 0x00007fb188359387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55

1 0x00007fb18835aa78 in __GI_abort () at abort.c:90

2 0x00007fb1883521a6 in __assert_fail_base (fmt=0x7fb1884adce0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",

assertion=assertion@entry=0x7fb187a0a985 "!*\"refcnt sub-zero\"", file=file@entry=0x7fb187a0a980 "rd.h", line=line@entry=326, 
function=function@entry=0x7fb187a13d39 <__PRETTY_FUNCTION__.5115> "rd_refcnt_sub0") at assert.c:92

3 0x00007fb188352252 in __GI___assert_fail (assertion=assertion@entry=0x7fb187a0a985 "!*\"refcnt sub-zero\"", file=file@entry=0x7fb187a0a980 "rd.h",

line=line@entry=326, function=function@entry=0x7fb187a13d39 <__PRETTY_FUNCTION__.5115> "rd_refcnt_sub0") at assert.c:101

4 0x00007fb187920646 in rd_refcnt_sub0 (R=0x7fb1780a8310) at rd.h:326

5 0x00007fb18794d0ff in rd_refcnt_sub0 (R=0x7fb1780a8310) at rdkafka_topic.h:171

6 rd_kafka_topic_destroy_app (app_rkt=0x7fb1780a8260) at rdkafka_topic.c:84

7 rd_kafka_topic_destroy (app_rkt=0x7fb1780a8260) at rdkafka_topic.c:129

8 0x00007fb18794ef1a in rd_kafka_topic_opaque (app_rkt=) at rdkafka_topic.c:1475

9 0x00007fb1876ff7db in RdKafka::MessageImpl::MessageImpl (rkmessage=0x7fb1780a9078, this=0x7fb154000a30) at rdkafkacpp_int.h:345

10 RdKafka::KafkaConsumerImpl::consume (this=, timeout_ms=) at KafkaConsumerImpl.cpp:119

11 0x00007fb189d10c9d in des::kafka::KafkaReader::poll (this=0x10000c0, consumerTimeout=1000) at KafkaReader.C:491

12 0x00007fb189d26acf in execute_native_thread_routine () from /u/dhamodha/fps/lib/Linux.RHEL7.x86_64/libdesfps1.5.so

13 0x00007fb1886f8ea5 in start_thread (arg=0x7fb1673c1700) at pthread_create.c:307

14 0x00007fb1884218dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

dshivashankar1994 commented 4 years ago

Any updates on this @edenhill ? Is there anything else I can provide to you that would help?

edenhill commented 4 years ago

Make sure that you are destroying each Message and Topic object exactly once.

dshivashankar1994 commented 4 years ago

I am creating TopicPartition using RdKafka::TopicPartition::create(topic->topic(), partition->id()). But not clearing it at all. Also I had a delete message. Even after commenting out, I face the problem.

edenhill commented 4 years ago

Can you share your code?

dshivashankar1994 commented 4 years ago

I'll give you a minimal reproducer by tomorrow.

dshivashankar1994 commented 4 years ago

I am able to reproduce this using th rdkafka_complex_example_cpp itself. After the topic gets deleted, I was getting the error topic does not exist. As soon as the topic metadata refresh happens, I am getting the assertion error.

/codemill/dhamodha/librdkafka-master/examples/rdkafka_complex_consumer_example_cpp -b mwkafka-staging-01.nyc.deshaw.com:9092,mwkafka-staging-02.nyc.deshaw.com:9092,mwkafka-staging-01.dr.deshaw.com:9092,mwkafka-staging-02.dr.deshaw.com:9092,mwkafka-staging-01.tbd.deshaw.com:909 -g new_cons_gp -X security.protocol=SASL_PLAINTEXT -X sasl.kerberos.service.name=kafka -X sasl.mechanism=GSSAPI -X sasl.kerberos.kinit.cmd='skinit --force' -X client.id=cppcon fps.dhamodha.test_hello 
dhamodha acquired ticket: client dhamodha/hostbased@UNIX.DESHAW.COM, server krbtgt/UNIX.DESHAW.COM@UNIX.DESHAW.COM
% Created consumer cppcon#consumer-1
2020-07-07 10:04:45.922: LOG-3-FAIL: [thrd:sasl_plaintext://mwkafka-staging-01.tbd.deshaw.com:909/bootstra]: sasl_plaintext://mwkafka-staging-01.tbd.deshaw.com:909/bootstrap: Connect to ipv4#10.236.30.7:909 failed: Connection refused (after 206ms in state CONNECT)
2020-07-07 10:04:45.922: ERROR (Local: Broker transport failure): sasl_plaintext://mwkafka-staging-01.tbd.deshaw.com:909/bootstrap: Connect to ipv4#10.236.30.7:909 failed: Connection refused (after 206ms in state CONNECT)
RebalanceCb: Local: Assign partitions: fps.dhamodha.test_hello[0], 
2020-07-07 10:05:41.361: LOG-5-PARTCNT: [thrd:main]: Topic fps.dhamodha.test_hello partition count changed from 1 to 0
2020-07-07 10:05:41.362: ERROR (Local: Unknown partition): fps.dhamodha.test_hello [0]: desired partition no longer exists (Local: Unknown partition)
2020-07-07 10:05:41.362: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test_hello [0]: topic does not exist (Broker: Unknown topic or partition)
2020-07-07 10:05:41.921: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test_hello [0]: topic does not exist (Broker: Unknown topic or partition)
...
...
2020-07-07 10:09:44.920: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test_hello [0]: topic does not exist (Broker: Unknown topic or partition)
rdkafka_complex_consumer_example_cpp: rd.h:326: rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed.
Aborted (core dumped)

From core:

Core was generated by `/codemill/dhamodha/librdkafka-master/examples/rdkafka_complex_consumer_example_'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f95a794b387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55    return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
[Current thread is 1 (Thread 0x7f95a9cc07c0 (LWP 38127))]
(gdb) bt
#0  0x00007f95a794b387 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007f95a794ca78 in __GI_abort () at abort.c:90
#2  0x00007f95a79441a6 in __assert_fail_base (fmt=0x7f95a7a9fce0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x4ee839 "!*\"refcnt sub-zero\"", 
    file=file@entry=0x4ee834 "rd.h", line=line@entry=326, function=function@entry=0x4f7b59 <__PRETTY_FUNCTION__.5115> "rd_refcnt_sub0") at assert.c:92
#3  0x00007f95a7944252 in __GI___assert_fail (assertion=assertion@entry=0x4ee839 "!*\"refcnt sub-zero\"", file=file@entry=0x4ee834 "rd.h", line=line@entry=326, 
    function=function@entry=0x4f7b59 <__PRETTY_FUNCTION__.5115> "rd_refcnt_sub0") at assert.c:101
#4  0x00000000004086cb in rd_refcnt_sub0 (R=0x7f959c0026f0) at rd.h:326
#5  0x000000000043efaf in rd_refcnt_sub0 (R=0x7f959c0026f0) at rdkafka_topic.h:171
#6  rd_kafka_topic_destroy_app (app_rkt=0x7f959c002640) at rdkafka_topic.c:84
#7  rd_kafka_topic_destroy (app_rkt=0x7f959c002640) at rdkafka_topic.c:129
#8  0x0000000000440dca in rd_kafka_topic_opaque (app_rkt=<optimized out>) at rdkafka_topic.c:1475
#9  0x000000000040cf7b in RdKafka::MessageImpl::MessageImpl (rkmessage=0x7f959c003c58, this=0x255a250) at rdkafkacpp_int.h:345
#10 RdKafka::KafkaConsumerImpl::consume (this=<optimized out>, timeout_ms=<optimized out>) at KafkaConsumerImpl.cpp:119
#11 0x000000000040a111 in main (argc=<optimized out>, argv=<optimized out>) at rdkafka
edenhill commented 4 years ago

Is this with a completely unmodified v1.4.2 rdkafka_complex_consumer_example ? I can't reproduce it.

Can you try to reproduce on master?

dshivashankar1994 commented 4 years ago

Yes, in master also its reproducible. rdkafka_complex_consumer_example_cpp with no modification. After subscribing, deleted the topic - "fps.dhamodha.test.hello"

/codemill/dhamodha/librdkafka(master)> /codemill/dhamodha/librdkafka/examples/rdkafka_complex_consumer_example_cpp -b mwkafka-staging-01.nyc.firm.com:9092,mwkafka-staging-02.nyc.firm.com:9092,mwkafka-staging-01.dr.firm.com:9092,mwkafka-staging-02.dr.firm.com:9092,mwkafka-staging-01.tbd.firm.com:909 -g new_cons_gp -X security.protocol=SASL_PLAINTEXT -X sasl.kerberos.service.name=kafka -X sasl.mechanism=GSSAPI -X sasl.kerberos.kinit.cmd='skinit --force' -X client.id=cppcon fps.dhamodha.test.hello 
dhamodha acquired ticket: client dhamodha/hostbased@UNIX.firm.COM, server krbtgt/UNIX.firm.COM@UNIX.firm.COM
% Created consumer cppcon#consumer-1
2020-07-07 19:03:12.476: LOG-3-FAIL: [thrd:sasl_plaintext://mwkafka-staging-01.tbd.firm.com:909/bootstra]: sasl_plaintext://mwkafka-staging-01.tbd.firm.com:909/bootstrap: Connect to ipv4#10.236.30.7:909 failed: Connection refused (after 197ms in state CONNECT)
2020-07-07 19:03:12.476: ERROR (Local: Broker transport failure): sasl_plaintext://mwkafka-staging-01.tbd.firm.com:909/bootstrap: Connect to ipv4#10.236.30.7:909 failed: Connection refused (after 197ms in state CONNECT)
RebalanceCb: Local: Assign partitions: fps.dhamodha.test.hello[0], 
dhamodha acquired ticket: client dhamodha/hostbased@UNIX.firm.COM, server krbtgt/UNIX.firm.COM@UNIX.firm.COM
2020-07-07 19:04:23.228: LOG-5-PARTCNT: [thrd:main]: Topic fps.dhamodha.test.hello partition count changed from 1 to 0
2020-07-07 19:04:23.228: ERROR (Local: Unknown partition): fps.dhamodha.test.hello [0]: desired partition no longer exists (Local: Unknown partition)
2020-07-07 19:04:23.228: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test.hello [0]: topic does not exist (Broker: Unknown topic or partition)
2020-07-07 19:04:23.479: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test.hello [0]: topic does not exist (Broker: Unknown topic or partition)
...
...
2020-07-07 19:08:10.926: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test.hello [0]: topic does not exist (Broker: Unknown topic or partition)
2020-07-07 19:08:11.477: ERROR (Broker: Unknown topic or partition): fps.dhamodha.test.hello [0]: topic does not exist (Broker: Unknown topic or partition)
rdkafka_complex_consumer_example_cpp: rd.h:326: rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed.
Aborted (core dumped)
edenhill commented 4 years ago

Can you reproduce this with ASAN? (build librdkafka with ./dev-conf.sh asan)

dshivashankar1994 commented 4 years ago

I don't have libasan installed. I'm restricted from installing it due to regulations.

Are you unable to reproduce the error with the master ?

Please let me know if I can do anything else instead.

dshivashankar1994 commented 4 years ago

Hi, Any updates here ?

dshivashankar1994 commented 4 years ago

Ping

edenhill commented 4 years ago

Sorry, but I'm not able to reproduce this and I don't have much else to go on if it can't be run in a debugger/valgrind/asan.

dshivashankar1994 commented 4 years ago

If this is not reproducible with the same version of the rdkafka example, could it be because of different versions of the kafka broker ?

Also, please note that I have given the output of the debugger (gdb)

edenhill commented 4 years ago

What is the broker version?

dshivashankar1994 commented 4 years ago

Kafka broker version - 2.5.0.

To make sure, I just had this checkout, started kafka brokers, topic and producer. Started our consumer, deleted the topic and saw the above error (after few minutes)

The failure is not immediate. I was getting it after 4-5 minutes.

edenhill commented 4 years ago

Can't reproduce on master with AK 2.5.0. Start the consumer, wait 15 seconds, delete the second topic, let it run for 30 minutes, no crash.

dshivashankar1994 commented 4 years ago

Ran with ./dev-conf.sh asan and got the below error

amodha@sarkar01.nyc[psc]:~/version/librdkafka-1.5.0> ./dev-conf.sh asan
Development configuration options:  --enable-devel --disable-optimization
Removing Makefile.config
Removing config.h
Removing config.cache
Removing config.log
checking for OS or distribution... ok (redhatenterpriseserver)
checking for C compiler from CC env... failed
checking for gcc (by command)... ok
checking for C++ compiler from CXX env... failed
checking for C++ compiler (g++)... ok
checking executable ld... ok
checking executable nm... ok
checking executable objdump... ok
checking executable strip... ok
checking executable libtool... ok
checking for pkgconfig (by command)... ok
checking for install (by command)... ok
checking for PIC (by compile)... ok
checking for GNU-compatible linker options... ok
checking for GNU linker-script ld flag... ok
checking for __atomic_32 (by compile)... ok
checking for __atomic_64 (by compile)... ok
checking for socket (by compile)... ok
parsing version '0x010500ff'... ok (1.5.0)
checking for librt (by pkg-config)... failed
checking for librt (by compile)... ok
checking for libpthread (by pkg-config)... failed
checking for libpthread (by compile)... ok
checking for c11threads (by pkg-config)... failed
checking for c11threads (by compile)... failed (disable)
checking for libdl (by pkg-config)... failed
checking for libdl (by compile)... ok
checking for zlib (by pkg-config)... ok
checking for libcrypto (by pkg-config)... ok
checking for libssl (by pkg-config)... ok
checking for libsasl2 (by pkg-config)... ok
checking for libzstd (by pkg-config)... ok
checking for libm (by pkg-config)... failed
checking for libm (by compile)... ok
checking for liblz4 (by pkg-config)... ok
checking for syslog (by compile)... ok
checking for rapidjson (by compile)... ok
checking for crc32chw (by compile)... ok
checking for regex (by compile)... ok
checking for strndup (by compile)... ok
checking for strlcpy (by compile)... failed (disable)
checking for strerror_r (by compile)... ok
checking for pthread_setname_gnu (by compile)... ok
checking for nm (by env NM)... ok (cached)
checking for python3 (by command)... ok
checking for getrusage (by compile)... ok
Generated Makefile.config
Generated config.h

Configuration summary:
  prefix                   /usr/local
  MKL_DISTRO               redhatenterpriseserver
  SOLIB_EXT                .so
  ARCH                     x86_64
  CPU                      generic
  GEN_PKG_CONFIG           y
  MKL_APP_NAME             librdkafka
  MKL_APP_DESC_ONELINE     The Apache Kafka C/C++ library
  CC                       gcc
  CXX                      g++
  LD                       ld
  NM                       nm
  OBJDUMP                  objdump
  STRIP                    strip
  LIBTOOL                  libtool
  CPPFLAGS                  -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align
  LDFLAGS                  -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address
  PKG_CONFIG               pkg-config
  INSTALL                  install
  LIB_LDFLAGS              -shared -Wl,-soname,$(LIBFILENAME)
  LDFLAG_LINKERSCRIPT      -Wl,--version-script=
  RDKAFKA_VERSION_STR      1.5.0
  MKL_APP_VERSION          1.5.0
  LIBS                     -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
  CFLAGS                              
  CXXFLAGS                 -Wno-non-virtual-dtor
  SYMDUMPER                $(NM) -D
  MKL_DYNAMIC_LIBS         -llz4 -lm -lzstd -lsasl2 -lssl -lcrypto -lcrypto -lz -ldl -lpthread -lrt
  exec_prefix              /usr/local
  bindir                   /usr/local/bin
  sbindir                  /usr/local/sbin
  libexecdir               /usr/local/libexec
  datadir                  /usr/local/share
  sysconfdir               /usr/local/etc
  sharedstatedir           /usr/local/com
  localstatedir            /usr/local/var
  runstatedir              /usr/local/var/run
  libdir                   /usr/local/lib
  includedir               /usr/local/include
  infodir                  /usr/local/info
  mandir                   /usr/local/man
  BUILT_WITH               GCC GXX PKGCONFIG INSTALL GNULD LDS LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM LZ4_EXT SYSLOG RAPIDJSON SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW
Generated config.cache

Now type 'make' to build
make[1]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/tests'
rm -f *.test 0075-retry.o 0034-offset_reset.o 0084-destroy_flags.o 0006-symbols.o 0068-produce_timeout.o 0019-list_groups.o 0014-reconsume-191.o 0017-compression.o 0077-compaction.o 0031-get_offsets.o 0009-mock_cluster.o 0076-produce_retry.o 0003-msgmaxsize.o 0107-topic_recreate.o 0056-balanced_group_mt.o 0043-no_connection.o 0001-multiobj.o 0007-autotopic.o 0099-commit_metadata.o 0064-interceptors.o 0091-max_poll_interval_timeout.o 0021-rkt_destroy.o 0011-produce_batch.o 0028-long_topicnames.o 0069-consumer_add_parts.o 0016-client_swname.o 0104-fetch_from_follower_mock.o 0089-max_poll_interval.o 0025-timers.o 0094-idempotence_msg_timeout.o 0086-purge.o 0080-admin_ut.o 0033-regex_subscribe.o 0029-assign_offset.o 0038-performance.o 0079-fork.o 0090-idempotence.o 0055-producer_latency.o 0030-offset_commit.o 0036-partial_fetch.o 0050-subscribe_adds.o 0022-consume_batch.o 0020-destroy_hang.o 0048-partitioner.o 0081-admin.o 0088-produce_metadata_timeout.o 0051-assign_adds.o 0040-io_event.o 0002-unkpart.o 0093-holb.o 0103-transactions.o 0044-partition_cnt.o 0092-mixed_msgver.o 0004-conf.o 0112-assign_unknown_part.o 0013-null-msgs.o 0037-destroy_hang_local.o 0083-cb_event.o 0008-reqacks.o 0072-headers_ut.o 0052-msg_timestamps.o 0046-rkt_cache.o 0102-static_group_rebalance.o 0047-partial_buf_tmout.o 0026-consume_pause.o 0042-many_topics.o 0039-event.o 0005-order.o 0045-subscribe_update.o 0015-offset_seeks.o 0105-transactions_mock.o 0049-consume_conn_close.o 0018-cgrp_term.o 0000-unittests.o 0073-headers.o 0041-fetch_max_bytes.o 0062-stats_event.o 0012-produce_consume.o 0035-api_version.o 0106-cgrp_sess_timeout.o 0074-producev.o 0066-plugins.o 0061-consumer_lag.o 0110-batch_size.o 0059-bsearch.o 0085-headers.o 0109-auto_create_topics.o 0082-fetch_max_bytes.o 0098-consumer-txn.o 0101-fetch-from-follower.o 0067-empty_topic.o 0111-delay_create_topics.o 0097-ssl_verify.o 0060-op_prio.o 0065-yield.o 0058-log.o 0095-all_brokers_down.o 0063-clusterid.o 0054-offset_time.o 0070-null_empty.o 0100-thread_interceptors.o 0053-stats_cb.o 0057-invalid_topic.o 8000-idle.o 0078-c_from_cpp.o test.o rusage.o testcpp.o tinycthread.o tinycthread_extra.o rdlist.o sockem.o sockem_ctrl.o test-runner
make -C interceptor_test clean
make[2]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/tests/interceptor_test'
rm -f interceptor_test.o interceptor_test.d
rm -f interceptor_test.pc interceptor_test-static.pc
rm -f interceptor_test*.a interceptor_test.so.1 interceptor_test.so \
    interceptor_test.lds
make[2]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/tests/interceptor_test'
make[1]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/tests'
make[1]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/examples'
rm -f rdkafka_example rdkafka_performance rdkafka_example_cpp rdkafka_complex_consumer_example rdkafka_complex_consumer_example_cpp kafkatest_verifiable_client producer consumer idempotent_producer transactions
make[1]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/examples'
make[1]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/src'
rm -f rdkafka.o rdkafka_broker.o rdkafka_msg.o rdkafka_topic.o rdkafka_conf.o rdkafka_timer.o rdkafka_offset.o rdkafka_transport.o rdkafka_buf.o rdkafka_queue.o rdkafka_op.o rdkafka_request.o rdkafka_cgrp.o rdkafka_pattern.o rdkafka_partition.o rdkafka_subscription.o rdkafka_assignor.o rdkafka_range_assignor.o rdkafka_roundrobin_assignor.o rdkafka_feature.o rdcrc32.o crc32c.o rdmurmur2.o rdfnv1a.o rdaddr.o rdrand.o rdlist.o tinycthread.o tinycthread_extra.o rdlog.o rdstring.o rdkafka_event.o rdkafka_metadata.o rdregex.o rdports.o rdkafka_metadata_cache.o rdavl.o rdkafka_sasl.o rdkafka_sasl_plain.o rdkafka_interceptor.o rdkafka_msgset_writer.o rdkafka_msgset_reader.o rdkafka_header.o rdkafka_admin.o rdkafka_aux.o rdkafka_background.o rdkafka_idempotence.o rdkafka_cert.o rdkafka_txnmgr.o rdkafka_coord.o rdvarint.o rdbuf.o rdunittest.o rdkafka_mock.o rdkafka_mock_handlers.o rdkafka_mock_cgrp.o rdkafka_error.o rdkafka_sasl_cyrus.o rdkafka_sasl_scram.o rdkafka_sasl_oauthbearer.o snappy.o rdgz.o rdkafka_zstd.o rdhdrhistogram.o rdkafka_ssl.o rdkafka_lz4.o rdxxhash.o rddl.o rdkafka_plugin.o rdkafka.d rdkafka_broker.d rdkafka_msg.d rdkafka_topic.d rdkafka_conf.d rdkafka_timer.d rdkafka_offset.d rdkafka_transport.d rdkafka_buf.d rdkafka_queue.d rdkafka_op.d rdkafka_request.d rdkafka_cgrp.d rdkafka_pattern.d rdkafka_partition.d rdkafka_subscription.d rdkafka_assignor.d rdkafka_range_assignor.d rdkafka_roundrobin_assignor.d rdkafka_feature.d rdcrc32.d crc32c.d rdmurmur2.d rdfnv1a.d rdaddr.d rdrand.d rdlist.d tinycthread.d tinycthread_extra.d rdlog.d rdstring.d rdkafka_event.d rdkafka_metadata.d rdregex.d rdports.d rdkafka_metadata_cache.d rdavl.d rdkafka_sasl.d rdkafka_sasl_plain.d rdkafka_interceptor.d rdkafka_msgset_writer.d rdkafka_msgset_reader.d rdkafka_header.d rdkafka_admin.d rdkafka_aux.d rdkafka_background.d rdkafka_idempotence.d rdkafka_cert.d rdkafka_txnmgr.d rdkafka_coord.d rdvarint.d rdbuf.d rdunittest.d rdkafka_mock.d rdkafka_mock_handlers.d rdkafka_mock_cgrp.d rdkafka_error.d rdkafka_sasl_cyrus.d rdkafka_sasl_scram.d rdkafka_sasl_oauthbearer.d snappy.d rdgz.d rdkafka_zstd.d rdhdrhistogram.d rdkafka_ssl.d rdkafka_lz4.d rdxxhash.d rddl.d rdkafka_plugin.d
rm -f rdkafka.pc rdkafka-static.pc
rm -f librdkafka*.a librdkafka.so.1 librdkafka.so \
    librdkafka.lds
make[1]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/src'
make[1]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/src-cpp'
rm -f RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o RdKafka.d ConfImpl.d HandleImpl.d ConsumerImpl.d ProducerImpl.d KafkaConsumerImpl.d TopicImpl.d TopicPartitionImpl.d MessageImpl.d HeadersImpl.d QueueImpl.d MetadataImpl.d
rm -f rdkafka++.pc rdkafka++-static.pc
rm -f librdkafka++*.a librdkafka++.so.1 librdkafka++.so \

make[1]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/src-cpp'
make[1]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/src'
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka.c -o rdkafka.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_broker.c -o rdkafka_broker.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_msg.c -o rdkafka_msg.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_topic.c -o rdkafka_topic.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_conf.c -o rdkafka_conf.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_timer.c -o rdkafka_timer.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_offset.c -o rdkafka_offset.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_transport.c -o rdkafka_transport.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_buf.c -o rdkafka_buf.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_queue.c -o rdkafka_queue.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_op.c -o rdkafka_op.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_request.c -o rdkafka_request.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_cgrp.c -o rdkafka_cgrp.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_pattern.c -o rdkafka_pattern.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_partition.c -o rdkafka_partition.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_subscription.c -o rdkafka_subscription.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_assignor.c -o rdkafka_assignor.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_range_assignor.c -o rdkafka_range_assignor.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_roundrobin_assignor.c -o rdkafka_roundrobin_assignor.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_feature.c -o rdkafka_feature.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdcrc32.c -o rdcrc32.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c crc32c.c -o crc32c.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdmurmur2.c -o rdmurmur2.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdfnv1a.c -o rdfnv1a.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdaddr.c -o rdaddr.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdrand.c -o rdrand.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdlist.c -o rdlist.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c tinycthread.c -o tinycthread.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c tinycthread_extra.c -o tinycthread_extra.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdlog.c -o rdlog.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdstring.c -o rdstring.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_event.c -o rdkafka_event.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_metadata.c -o rdkafka_metadata.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdregex.c -o rdregex.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdports.c -o rdports.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_metadata_cache.c -o rdkafka_metadata_cache.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdavl.c -o rdavl.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_sasl.c -o rdkafka_sasl.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_sasl_plain.c -o rdkafka_sasl_plain.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_interceptor.c -o rdkafka_interceptor.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_msgset_writer.c -o rdkafka_msgset_writer.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_msgset_reader.c -o rdkafka_msgset_reader.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_header.c -o rdkafka_header.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_admin.c -o rdkafka_admin.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_aux.c -o rdkafka_aux.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_background.c -o rdkafka_background.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_idempotence.c -o rdkafka_idempotence.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_cert.c -o rdkafka_cert.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_txnmgr.c -o rdkafka_txnmgr.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_coord.c -o rdkafka_coord.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdvarint.c -o rdvarint.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdbuf.c -o rdbuf.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdunittest.c -o rdunittest.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_mock.c -o rdkafka_mock.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_mock_handlers.c -o rdkafka_mock_handlers.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_mock_cgrp.c -o rdkafka_mock_cgrp.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_error.c -o rdkafka_error.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_sasl_cyrus.c -o rdkafka_sasl_cyrus.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_sasl_scram.c -o rdkafka_sasl_scram.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_sasl_oauthbearer.c -o rdkafka_sasl_oauthbearer.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c snappy.c -o snappy.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdgz.c -o rdgz.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_zstd.c -o rdkafka_zstd.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdhdrhistogram.c -o rdhdrhistogram.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_ssl.c -o rdkafka_ssl.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_lz4.c -o rdkafka_lz4.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -O3 -c rdxxhash.c -o rdxxhash.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rddl.c -o rddl.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -c rdkafka_plugin.c -o rdkafka_plugin.o
Generating linker script librdkafka.lds from rdkafka.h rdkafka_mock.h
Generating pkg-config file rdkafka.pc
Generating pkg-config file rdkafka-static.pc
Creating shared library librdkafka.so.1
gcc -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -shared -Wl,-soname,librdkafka.so.1 -Wl,--version-script=librdkafka.lds rdkafka.o rdkafka_broker.o rdkafka_msg.o rdkafka_topic.o rdkafka_conf.o rdkafka_timer.o rdkafka_offset.o rdkafka_transport.o rdkafka_buf.o rdkafka_queue.o rdkafka_op.o rdkafka_request.o rdkafka_cgrp.o rdkafka_pattern.o rdkafka_partition.o rdkafka_subscription.o rdkafka_assignor.o rdkafka_range_assignor.o rdkafka_roundrobin_assignor.o rdkafka_feature.o rdcrc32.o crc32c.o rdmurmur2.o rdfnv1a.o rdaddr.o rdrand.o rdlist.o tinycthread.o tinycthread_extra.o rdlog.o rdstring.o rdkafka_event.o rdkafka_metadata.o rdregex.o rdports.o rdkafka_metadata_cache.o rdavl.o rdkafka_sasl.o rdkafka_sasl_plain.o rdkafka_interceptor.o rdkafka_msgset_writer.o rdkafka_msgset_reader.o rdkafka_header.o rdkafka_admin.o rdkafka_aux.o rdkafka_background.o rdkafka_idempoteCreating static library librdkafka.a
nce.o rdkafka_cert.o rdkafka_txnmgr.o rdkafka_coord.o rdvarint.o rdbuf.o rdunittest.o rdkafka_mock.o rdkafka_mock_handlers.o rdkafka_mock_cgrp.o rdkafka_error.o rdkafka_sasl_cyrus.o rdkafka_sasl_scram.o rdkafka_sasl_oauthbearer.o snappy.o rdgz.o rdkafka_zstd.o rdhdrhistogram.o rdkafka_ssl.o rdkafka_lz4.o rdxxhash.o rddl.o rdkafka_plugin.o -o librdkafka.so.1 -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
ar rcs librdkafka.a rdkafka.o rdkafka_broker.o rdkafka_msg.o rdkafka_topic.o rdkafka_conf.o rdkafka_timer.o rdkafka_offset.o rdkafka_transport.o rdkafka_buf.o rdkafka_queue.o rdkafka_op.o rdkafka_request.o rdkafka_cgrp.o rdkafka_pattern.o rdkafka_partition.o rdkafka_subscription.o rdkafka_assignor.o rdkafka_range_assignor.o rdkafka_roundrobin_assignor.o rdkafka_feature.o rdcrc32.o crc32c.o rdmurmur2.o rdfnv1a.o rdaddr.o rdrand.o rdlist.o tinycthread.o tinycthread_extra.o rdlog.o rdstring.o rdkafka_event.o rdkafka_metadata.o rdregex.o rdports.o rdkafka_metadata_cache.o rdavl.o rdkafka_sasl.o rdkafka_sasl_plain.o rdkafka_interceptor.o rdkafka_msgset_writer.o rdkafka_msgset_reader.o rdkafka_header.o rdkafka_admin.o rdkafka_aux.o rdkafka_background.o rdkafka_idempotence.o rdkafka_cert.o rdkafka_txnmgr.o rdkafka_coord.o rdvarint.o rdbuf.o rdunittest.o rdkafka_mock.o rdkafka_mock_handlers.o rdkafka_mock_cgrp.o rdkafka_error.o rdkafka_sasl_cyrus.o rdkafka_sasl_scram.o rdkafka_sasl_oauthbearer.o snappy.o rdgz.o rdkafka_zstd.o rdhdrhistogram.o rdkafka_ssl.o rdkafka_lz4.o rdxxhash.o rddl.o rdkafka_plugin.o
Creating librdkafka.so symlink
rm -f "librdkafka.so" && ln -s "librdkafka.so.1" "librdkafka.so"
Checking librdkafka integrity
librdkafka.so.1                OK
librdkafka.a                   OK
Symbol visibility              OK
make[1]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/src'
make[1]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/src-cpp'
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c RdKafka.cpp -o RdKafka.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c ConfImpl.cpp -o ConfImpl.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c HandleImpl.cpp -o HandleImpl.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c ConsumerImpl.cpp -o ConsumerImpl.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c ProducerImpl.cpp -o ProducerImpl.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c KafkaConsumerImpl.cpp -o KafkaConsumerImpl.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c TopicImpl.cpp -o TopicImpl.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c TopicPartitionImpl.cpp -o TopicPartitionImpl.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c MessageImpl.cpp -o MessageImpl.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c HeadersImpl.cpp -o HeadersImpl.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c QueueImpl.cpp -o QueueImpl.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c MetadataImpl.cpp -o MetadataImpl.o
Generating pkg-config file rdkafka++.pc
Generating pkg-config file rdkafka++-static.pc
Creating shared library librdkafka++.so.1
Creating static library librdkafka++.a
g++ -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -shared -Wl,-soname,librdkafka++.so.1 RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o -o librdkafka++.so.1 -L../src -lrdkafka
ar rcs librdkafka++.a RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o
Creating librdkafka++.so symlink
rm -f "librdkafka++.so" && ln -s "librdkafka++.so.1" "librdkafka++.so"
Checking librdkafka++ integrity
librdkafka++.so.1              OK
librdkafka++.a                 OK
make[1]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/src-cpp'
make -C examples
make[1]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/examples'
gcc  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src rdkafka_example.c -o rdkafka_example -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address \
    ../src/librdkafka.a -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
gcc  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src rdkafka_performance.c -o rdkafka_performance -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address \
    ../src/librdkafka.a -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
g++  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp rdkafka_example.cpp -o rdkafka_example_cpp -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address \
    ../src-cpp/librdkafka++.a ../src/librdkafka.a -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
gcc  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src rdkafka_complex_consumer_example.c -o rdkafka_complex_consumer_example -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address \
    ../src/librdkafka.a -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
g++  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp rdkafka_complex_consumer_example.cpp -o rdkafka_complex_consumer_example_cpp -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address \
    ../src-cpp/librdkafka++.a ../src/librdkafka.a -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
g++  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp kafkatest_verifiable_client.cpp -o kafkatest_verifiable_client -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address \
    ../src-cpp/librdkafka++.a ../src/librdkafka.a -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
gcc  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src producer.c -o producer -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address \
    ../src/librdkafka.a -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
gcc  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src consumer.c -o consumer -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address \
    ../src/librdkafka.a -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
gcc  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src idempotent_producer.c -o idempotent_producer -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address \
    ../src/librdkafka.a -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
gcc  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align  -I../src transactions.c -o transactions -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address \
    ../src/librdkafka.a -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt
# rdkafka_complex_consumer_example is ready
#
./rdkafka_complex_consumer_example <topic[:part]> <topic2[:part]> ..

#
# More usage options:
./rdkafka_complex_consumer_example -h
# rdkafka_example is ready
#
# Run producer (write messages on stdin)
./rdkafka_example -P -t <topic> -p <partition>

# or consumer
./rdkafka_example -C -t <topic> -p <partition>

#
# More usage options:
./rdkafka_example -h
# rdkafka_performance is ready
#
# Run producer
./rdkafka_performance -P -t <topic> -p <partition> -s <msgsize>

# or consumer
./rdkafka_performance -C -t <topic> -p <partition>

#
# More usage options:
./rdkafka_performance -h
make[1]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/examples'
Updating CONFIGURATION.md
cmp: EOF on CONFIGURATION.md
Checking  integrity
CONFIGURATION.md               OK
examples/rdkafka_example       OK
examples/rdkafka_performance   OK
examples/rdkafka_example_cpp   OK
make[1]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/src'
Generating pkg-config file rdkafka-static.pc
Checking librdkafka integrity
librdkafka.so.1                OK
librdkafka.a                   OK
Symbol visibility              OK
make[1]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/src'
make[1]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/src-cpp'
Generating pkg-config file rdkafka++-static.pc
Checking librdkafka++ integrity
librdkafka++.so.1              OK
librdkafka++.a                 OK
make[1]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/src-cpp'
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0075-retry.c -o 0075-retry.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0042-many_topics.c -o 0042-many_topics.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0034-offset_reset.c -o 0034-offset_reset.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0029-assign_offset.c -o 0029-assign_offset.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0022-consume_batch.c -o 0022-consume_batch.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0033-regex_subscribe.c -o 0033-regex_subscribe.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0084-destroy_flags.c -o 0084-destroy_flags.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0020-destroy_hang.c -o 0020-destroy_hang.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0014-reconsume-191.c -o 0014-reconsume-191.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0008-reqacks.c -o 0008-reqacks.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0038-performance.c -o 0038-performance.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0021-rkt_destroy.c -o 0021-rkt_destroy.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0011-produce_batch.c -o 0011-produce_batch.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0049-consume_conn_close.c -o 0049-consume_conn_close.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0009-mock_cluster.c -o 0009-mock_cluster.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0028-long_topicnames.c -o 0028-long_topicnames.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0045-subscribe_update.c -o 0045-subscribe_update.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0076-produce_retry.c -o 0076-produce_retry.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0069-consumer_add_parts.c -o 0069-consumer_add_parts.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0016-client_swname.c -o 0016-client_swname.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0091-max_poll_interval_timeout.c -o 0091-max_poll_interval_timeout.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0107-topic_recreate.c -o 0107-topic_recreate.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0012-produce_consume.c -o 0012-produce_consume.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0015-offset_seeks.c -o 0015-offset_seeks.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0081-admin.c -o 0081-admin.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0104-fetch_from_follower_mock.c -o 0104-fetch_from_follower_mock.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0056-balanced_group_mt.c -o 0056-balanced_group_mt.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0088-produce_metadata_timeout.c -o 0088-produce_metadata_timeout.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0062-stats_event.c -o 0062-stats_event.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0079-fork.c -o 0079-fork.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0043-no_connection.c -o 0043-no_connection.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0006-symbols.c -o 0006-symbols.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0099-commit_metadata.c -o 0099-commit_metadata.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0090-idempotence.c -o 0090-idempotence.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0064-interceptors.c -o 0064-interceptors.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0089-max_poll_interval.c -o 0089-max_poll_interval.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0048-partitioner.c -o 0048-partitioner.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0025-timers.c -o 0025-timers.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0072-headers_ut.c -o 0072-headers_ut.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0002-unkpart.c -o 0002-unkpart.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0105-transactions_mock.c -o 0105-transactions_mock.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0052-msg_timestamps.c -o 0052-msg_timestamps.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0093-holb.c -o 0093-holb.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0055-producer_latency.c -o 0055-producer_latency.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0103-transactions.c -o 0103-transactions.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0030-offset_commit.c -o 0030-offset_commit.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0018-cgrp_term.c -o 0018-cgrp_term.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0044-partition_cnt.c -o 0044-partition_cnt.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0031-get_offsets.c -o 0031-get_offsets.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0000-unittests.c -o 0000-unittests.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0035-api_version.c -o 0035-api_version.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0073-headers.c -o 0073-headers.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0036-partial_fetch.c -o 0036-partial_fetch.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0092-mixed_msgver.c -o 0092-mixed_msgver.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0106-cgrp_sess_timeout.c -o 0106-cgrp_sess_timeout.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0046-rkt_cache.c -o 0046-rkt_cache.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0102-static_group_rebalance.c -o 0102-static_group_rebalance.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0068-produce_timeout.c -o 0068-produce_timeout.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0004-conf.c -o 0004-conf.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0050-subscribe_adds.c -o 0050-subscribe_adds.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0074-producev.c -o 0074-producev.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0112-assign_unknown_part.c -o 0112-assign_unknown_part.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0051-assign_adds.c -o 0051-assign_adds.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0039-event.c -o 0039-event.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0094-idempotence_msg_timeout.c -o 0094-idempotence_msg_timeout.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0040-io_event.c -o 0040-io_event.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0005-order.c -o 0005-order.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0086-purge.c -o 0086-purge.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0001-multiobj.c -o 0001-multiobj.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0080-admin_ut.c -o 0080-admin_ut.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0019-list_groups.c -o 0019-list_groups.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0013-null-msgs.c -o 0013-null-msgs.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0007-autotopic.c -o 0007-autotopic.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0041-fetch_max_bytes.c -o 0041-fetch_max_bytes.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0047-partial_buf_tmout.c -o 0047-partial_buf_tmout.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0037-destroy_hang_local.c -o 0037-destroy_hang_local.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0026-consume_pause.c -o 0026-consume_pause.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0017-compression.c -o 0017-compression.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0083-cb_event.c -o 0083-cb_event.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0003-msgmaxsize.c -o 0003-msgmaxsize.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c 0077-compaction.c -o 0077-compaction.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0109-auto_create_topics.cpp -o 0109-auto_create_topics.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0067-empty_topic.cpp -o 0067-empty_topic.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0095-all_brokers_down.cpp -o 0095-all_brokers_down.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0060-op_prio.cpp -o 0060-op_prio.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0065-yield.cpp -o 0065-yield.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0058-log.cpp -o 0058-log.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 8000-idle.cpp -o 8000-idle.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0082-fetch_max_bytes.cpp -o 0082-fetch_max_bytes.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0110-batch_size.cpp -o 0110-batch_size.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0059-bsearch.cpp -o 0059-bsearch.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0098-consumer-txn.cpp -o 0098-consumer-txn.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0061-consumer_lag.cpp -o 0061-consumer_lag.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0063-clusterid.cpp -o 0063-clusterid.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0054-offset_time.cpp -o 0054-offset_time.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0111-delay_create_topics.cpp -o 0111-delay_create_topics.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0097-ssl_verify.cpp -o 0097-ssl_verify.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0101-fetch-from-follower.cpp -o 0101-fetch-from-follower.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0100-thread_interceptors.cpp -o 0100-thread_interceptors.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0078-c_from_cpp.cpp -o 0078-c_from_cpp.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0085-headers.cpp -o 0085-headers.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0053-stats_cb.cpp -o 0053-stats_cb.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0070-null_empty.cpp -o 0070-null_empty.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0057-invalid_topic.cpp -o 0057-invalid_topic.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c 0066-plugins.cpp -o 0066-plugins.o
make -C interceptor_test
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c rusage.c -o rusage.o
g++ -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src -I../src-cpp -Wno-non-virtual-dtor -c testcpp.cpp -o testcpp.o
gcc  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c ../src/tinycthread.c
gcc  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c ../src/tinycthread_extra.c
gcc  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c ../src/rdlist.c
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c sockem.c -o sockem.o
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c sockem_ctrl.c -o sockem_ctrl.o
make[1]: Entering directory `/u/dhamodha/version/librdkafka-1.5.0/tests/interceptor_test'
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../../src  -c interceptor_test.c -o interceptor_test.o
Generating pkg-config file interceptor_test.pc
Generating pkg-config file interceptor_test-static.pc
Creating shared library interceptor_test.so.1
Creating static library interceptor_test.a
gcc -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -rdynamic -L../src -L../src-cpp -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib -fsanitize=address -L../../src -shared -Wl,-soname,interceptor_test.so.1 interceptor_test.o -o interceptor_test.so.1 -llz4   -lm -lzstd   -lsasl2   -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt -lrdkafka
ar rcs interceptor_test.a interceptor_test.o
Creating interceptor_test.so symlink
rm -f "interceptor_test.so" && ln -s "interceptor_test.so.1" "interceptor_test.so"
make[1]: Leaving directory `/u/dhamodha/version/librdkafka-1.5.0/tests/interceptor_test'
gcc -MD -MP  -fsanitize=address -fsanitize=address -g -O0 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I../src  -c test.c -o test.o
0053-stats_cb.cpp: In function \u2018void test_stats()\u2019:
0053-stats_cb.cpp:413:47: internal compiler error: in gimplify_init_constructor, at gimplify.c:4271
   struct exp_part_stats exp_parts[partcnt] = {};
                                               ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugzilla.redhat.com/bugzilla> for instructions.
Preprocessed source stored into /tmp/ccntey5E.out file, please attach this to your bugreport.
make: *** [0053-stats_cb.o] Error 1
dshivashankar1994 commented 4 years ago

After doing ./dev-conf.sh asan, ran make and did the below steps and reproduced the problem.

tar -xzf kafka_2.12-2.5.0.tgz
cd kafka_2.12-2.5.0/
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test_1
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test_1

Produced few messages

Started consumer and left it running

dhamodha@sarkar01.nyc[psc]:~> /u/dhamodha/version/librdkafka-1.5.0/examples/rdkafka_complex_consumer_example_cpp -b localhost:9092 -g new_cons_gp test_1
% Created consumer rdkafka#consumer-1
RebalanceCb: Local: Assign partitions: test_1[0], 

msg 1
msg 2

From consumer output

2020-07-23 08:19:35.228: LOG-5-PARTCNT: [thrd:main]: Topic test_1 partition count changed from 1 to 0
2020-07-23 08:19:35.228: ERROR (Local: Unknown partition): test_1 [0]: desired partition no longer exists (Local: Unknown partition)
2020-07-23 08:19:35.228: ERROR (Broker: Unknown topic or partition): test_1 [0]: topic does not exist (Broker: Unknown topic or partition)
2020-07-23 08:19:35.431: ERROR (Broker: Unknown topic or partition): test_1 [0]: topic does not exist (Broker: Unknown topic or partition)
2020-07-23 08:19:36.431: ERROR (Broker: Unknown topic or partition): test_1 [0]: topic does not exist (Broker: Unknown topic or partition)
...
...
2020-07-23 08:24:01.429: ERROR (Broker: Unknown topic or partition): test_1 [0]: topic does not exist (Broker: Unknown topic or partition)
rdkafka_complex_consumer_example_cpp: rd.h:326: rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed.
Aborted

I ain't doing anything else.

dshivashankar1994 commented 4 years ago

Before using the example kafka, did this 

export LDFLAGS="-L/var/tmp/local/lib -Wl,-R/var/tmp/local/lib"
./configure --CPPFLAGS="-I/var/tmp/local/include"  --prefix /var/tmp/local 
./dev-conf.sh asan 
make

Let me know if I missing anything.

edenhill commented 4 years ago

This is on librdkafka master with AK 2.5.0:

% Subscribing to 2 topics
% Consumer group rebalanced: assigned:
 testa [0] offset -1001, testb [0] offset -1001
% Consumer reached end of testa [0] message queue at offset 10
% Consumer reached end of testb [0] message queue at offset 10
^Z
[1]+  Stopped                 examples/rdkafka_complex_consumer_example -g x2 -b $BROKERS testa testb
magnus@mbp:~/src/librdkafka (incrreb)$ bg
[1]+ examples/rdkafka_complex_consumer_example -g x2 -b $BROKERS testa testb &
magnus@mbp:~/src/librdkafka (incrreb)$ /Users/magnus/src/librdkafka/tests/tmp/LibrdkafkaTestCluster/KafkaBrokerApp/kafka/2.6.0-rc0/bin/kafka-topics.sh --bootstrap-server $BROKERS --delete --topic testb
1595512342.391 RDKAFKA-5-PARTCNT: rdkafka#consumer-1: [thrd:main]: Topic testb partition count changed from 1 to 0
1595512342.391 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: desired partition no longer exists (Local: Unknown partition)
1595512342.391 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: topic does not exist (Broker: Unknown topic or partition)
magnus@mbp:~/src/librdkafka (incrreb)$ 1595512343.198 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: topic does not exist (Broker: Unknown topic or partition)
1595512344.273 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: topic does not exist (Broker: Unknown topic or partition)
1595512345.279 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: topic does not exist (Broker: Unknown topic or partition)
1595512346.285 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: topic does not exist (Broker: Unknown topic or partition)
1595512347.291 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: topic does not exist (Broker: Unknown topic or partition)
1595512348.298 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: topic does not exist (Broker: Unknown topic or partition)
1595512349.305 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: topic does not exist (Broker: Unknown topic or partition)
...repeats...

1595512620.815 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: topic does not exist (Broker: Unknown topic or partition)
1595512621.319 RDKAFKA-3-ERROR: rdkafka#consumer-1: [thrd:app]: rdkafka#consumer-1: testb [0]: topic does not exist (Broker: Unknown topic or partition)
% Consume error for topic "testb" [0] offset -1001: Subscribed topic not available: testb: Broker: Unknown topic or partition
% Consumer group rebalanced: revoked:
 testa [0] offset -1001, testb [0] offset -1001
% Consumer group rebalanced: assigned:
 testa [0] offset -1001
% Consumer reached end of testa [0] message queue at offset 10

No crash.

dshivashankar1994 commented 4 years ago

Any thoughts on how to proceed ?

Also, please see if I am using asan in right way.

edenhill commented 4 years ago

It looks like it does build with asan correctly (the weird compiler bug can be ignored since that is in a test).

edenhill commented 4 years ago

Do you think you could make a docker image with the client so that we're testing the exact same build? Please leave the source in the docker image

dshivashankar1994 commented 4 years ago

I have uploaded the docker image, the docker file used and the consumer output here

dshivashankar1994 commented 4 years ago

Hi, let me know if the docker was useful.

dshivashankar1994 commented 4 years ago

Hi, Any updates here ?

dshivashankar1994 commented 4 years ago

Ping

edenhill commented 4 years ago

Thank you! The docker build reproduces the issue and I'll start looking into it 👍

edenhill commented 4 years ago

Fixed on master.

Thanks @dshivashankar1994 for putting in all the effort to provide a reproducible test case, very valuable!