Ubuntu 20.04 - undefined reference to `rd_kafka_incremental_assign'

[adammpolak]

When running make and attempting to link to executable kafka-unit-test

It cannot seem to find:

undefined reference to `rd_kafka_incremental_assign'

undefined reference to `rd_kafka_rebalance_protocol'

~/dev/modern-cpp-kafka/build$ make
Scanning dependencies of target kafka-unit-test
[  2%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestBrokerMetadata.cc.o
[  4%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestConsumerRecord.cc.o
[  7%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestError.cc.o
[  9%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestHeader.cc.o
[ 11%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestKafkaClientDefaultProperties.cc.o
[ 14%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestKafkaException.cc.o
[ 16%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestKafkaMetrics.cc.o
[ 19%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestProducerRecord.cc.o
[ 21%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestProperties.cc.o
[ 23%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestTimestamp.cc.o
[ 26%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestTypes.cc.o
[ 28%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestUnorderedOffsetCommitQueue.cc.o
[ 30%] Building CXX object tests/unit/CMakeFiles/kafka-unit-test.dir/TestUtility.cc.o
[ 33%] Linking CXX executable kafka-unit-test
/usr/bin/ld: CMakeFiles/kafka-unit-test.dir/TestKafkaClientDefaultProperties.cc.o: in function `kafka::clients::KafkaConsumer::changeAssignment(kafka::clients::KafkaConsumer::PartitionsRebalanceEvent, std::set<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int>, std::less<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int> >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int> > > const&)':
TestKafkaClientDefaultProperties.cc:(.text._ZN5kafka7clients13KafkaConsumer16changeAssignmentENS1_24PartitionsRebalanceEventERKSt3setISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiESt4lessISB_ESaISB_EE[_ZN5kafka7clients13KafkaConsumer16changeAssignmentENS1_24PartitionsRebalanceEventERKSt3setISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiESt4lessISB_ESaISB_EE]+0x1cf): undefined reference to `rd_kafka_incremental_assign'
/usr/bin/ld: TestKafkaClientDefaultProperties.cc:(.text._ZN5kafka7clients13KafkaConsumer16changeAssignmentENS1_24PartitionsRebalanceEventERKSt3setISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiESt4lessISB_ESaISB_EE[_ZN5kafka7clients13KafkaConsumer16changeAssignmentENS1_24PartitionsRebalanceEventERKSt3setISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEiESt4lessISB_ESaISB_EE]+0x38f): undefined reference to `rd_kafka_incremental_unassign'
/usr/bin/ld: CMakeFiles/kafka-unit-test.dir/TestKafkaClientDefaultProperties.cc.o: in function `kafka::clients::KafkaConsumer::onRebalance(rd_kafka_resp_err_t, rd_kafka_topic_partition_list_s*)':
TestKafkaClientDefaultProperties.cc:(.text._ZN5kafka7clients13KafkaConsumer11onRebalanceE19rd_kafka_resp_err_tP31rd_kafka_topic_partition_list_s[_ZN5kafka7clients13KafkaConsumer11onRebalanceE19rd_kafka_resp_err_tP31rd_kafka_topic_partition_list_s]+0xfe): undefined reference to `rd_kafka_rebalance_protocol'
collect2: error: ld returned 1 exit status
make[2]: *** [tests/unit/CMakeFiles/kafka-unit-test.dir/build.make:264: tests/unit/kafka-unit-test] Error 1
make[1]: *** [CMakeFiles/Makefile2:1013: tests/unit/CMakeFiles/kafka-unit-test.dir/all] Error 2
make: *** [Makefile:141: all] Error 2

[adammpolak]

@kenneth-jia

Is this an issue with linking to the librdkafka-dev library? https://github.com/edenhill/librdkafka/issues/95

[kenneth-jia]

No, @adammpolak The link errors mean you're using an old librdkafka release which doesn't support "incremental assignment", -- i.e. older than https://github.com/edenhill/librdkafka/releases/tag/v1.6.0 Please manually install the latest librdkafka release! Here's an example for installation, https://github.com/morganstanley/modern-cpp-kafka/blob/main/.github/workflows/kafka_api_ci_tests.yml#L132

[adammpolak]

@kenneth-jia I will give this a shot thank you

[adammpolak]

Following that install path worked thank you @kenneth-jia

It is odd because following the confluent install path made it seem like I had 1.9:

~/dev/modern-cpp-kafka/build$ sudo apt-get install librdkafka-dev
[sudo] password for adam: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
librdkafka-dev is already the newest version (1.9.0-1.cflt~ubu20).
0 upgraded, 0 newly installed, 0 to remove and 41 not upgraded.

Not sure why that did not work ^