search

confluentinc / confluent-kafka-python

Confluent's Kafka Python Client
http://docs.confluent.io/current/clients/confluent-kafka-python
Other
124 stars 897 forks source link

Building from source leads to undefined symbols #438

Closed mmgutsche closed 6 years ago

mmgutsche commented 6 years ago

Description

On Fedora 27 I try to get confluent-kafka-python running by building librdkafka from source as well as confluent-kafka-python. I chose the installation from source to get libsasl2 working. Building and installing seems to work well, but on import confluent_kafka I receive

/home/mgutsche/anaconda3/envs/vavel_env/lib/python3.6/site-packages/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so: undefined symbol: rd_kafka_topic_result_name Using the latest master from both repositories, e.g. 0.11.5

How to reproduce

git clone https://github.com/edenhill/librdkafka.git # install librdkafka
./configure # checking for libsasl2 (by pkg-config)... ok
make
sudo make install
git clone https://github.com/confluentinc/confluent-kafka-python.git # install confluent-kafka-python
python setup.py install
python -c "import confluent_kafka"
edenhill commented 6 years ago

is rd_kafka_topic_result_name the only undefined symbol reported? Can you paste the output of ldd /home/mgutsche/anaconda3/envs/vavel_env/lib/python3.6/site-packages/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so followed by ldd ..the-path-to/librdkafka.so.1(as reported by the first command)

mmgutsche commented 6 years ago

The funny thing is, librdkafka is not shown at all when I run ldd (I reinstalled just to be sure I did no mistake there, and somehow the installation directory changed):

ldd /home/mgutsche/anaconda3/envs/vavel_env/lib/python3.6/site-packages/confluent_kafka-0.11.5-py3.6-linux-x86_64.egg/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007ffdfa5df000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa58c930000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fa58c57a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa58ce0d000)

Here the install protocol of running python setup.py install


[~/confluent-kafka-python]$ pip uninstall confluent-kafka 
Uninstalling confluent-kafka-0.11.5:
  Would remove:
    /home/mgutsche/anaconda3/envs/vavel_env/lib/python3.6/site-packages/confluent_kafka-0.11.5-py3.6-linux-x86_64.egg
Proceed (y/n)? y
  Successfully uninstalled confluent-kafka-0.11.5
[~/confluent-kafka-python]$ rm -rf build/
[~/confluent-kafka-python]$ C_INCLUDE_PATH=/usr/local/include/librdkafka LIBRARY_PATH=/usr/local/lib/ python setup.py install
running install
running bdist_egg
running egg_info
writing confluent_kafka.egg-info/PKG-INFO
writing dependency_links to confluent_kafka.egg-info/dependency_links.txt
writing requirements to confluent_kafka.egg-info/requires.txt
writing top-level names to confluent_kafka.egg-info/top_level.txt
reading manifest file 'confluent_kafka.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'confluent_kafka.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/confluent_kafka
copying confluent_kafka/__init__.py -> build/lib.linux-x86_64-3.6/confluent_kafka
creating build/lib.linux-x86_64-3.6/confluent_kafka/avro
copying confluent_kafka/avro/error.py -> build/lib.linux-x86_64-3.6/confluent_kafka/avro
copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.linux-x86_64-3.6/confluent_kafka/avro
copying confluent_kafka/avro/load.py -> build/lib.linux-x86_64-3.6/confluent_kafka/avro
copying confluent_kafka/avro/__init__.py -> build/lib.linux-x86_64-3.6/confluent_kafka/avro
creating build/lib.linux-x86_64-3.6/confluent_kafka/admin
copying confluent_kafka/admin/__init__.py -> build/lib.linux-x86_64-3.6/confluent_kafka/admin
creating build/lib.linux-x86_64-3.6/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-3.6/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/__init__.py -> build/lib.linux-x86_64-3.6/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-3.6/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-3.6/confluent_kafka/kafkatest
creating build/lib.linux-x86_64-3.6/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-3.6/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/__init__.py -> build/lib.linux-x86_64-3.6/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/confluent_kafka
creating build/temp.linux-x86_64-3.6/confluent_kafka/src
gcc -pthread -B /home/mgutsche/anaconda3/envs/vavel_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mgutsche/anaconda3/envs/vavel_env/include/python3.6m -c confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-3.6/confluent_kafka/src/confluent_kafka.o
gcc -pthread -B /home/mgutsche/anaconda3/envs/vavel_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mgutsche/anaconda3/envs/vavel_env/include/python3.6m -c confluent_kafka/src/Producer.c -o build/temp.linux-x86_64-3.6/confluent_kafka/src/Producer.o
gcc -pthread -B /home/mgutsche/anaconda3/envs/vavel_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mgutsche/anaconda3/envs/vavel_env/include/python3.6m -c confluent_kafka/src/Consumer.c -o build/temp.linux-x86_64-3.6/confluent_kafka/src/Consumer.o
gcc -pthread -B /home/mgutsche/anaconda3/envs/vavel_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mgutsche/anaconda3/envs/vavel_env/include/python3.6m -c confluent_kafka/src/Metadata.c -o build/temp.linux-x86_64-3.6/confluent_kafka/src/Metadata.o
gcc -pthread -B /home/mgutsche/anaconda3/envs/vavel_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mgutsche/anaconda3/envs/vavel_env/include/python3.6m -c confluent_kafka/src/AdminTypes.c -o build/temp.linux-x86_64-3.6/confluent_kafka/src/AdminTypes.o
gcc -pthread -B /home/mgutsche/anaconda3/envs/vavel_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mgutsche/anaconda3/envs/vavel_env/include/python3.6m -c confluent_kafka/src/Admin.c -o build/temp.linux-x86_64-3.6/confluent_kafka/src/Admin.o
gcc -pthread -shared -B /home/mgutsche/anaconda3/envs/vavel_env/compiler_compat -L/home/mgutsche/anaconda3/envs/vavel_env/lib -Wl,-rpath=/home/mgutsche/anaconda3/envs/vavel_env/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/confluent_kafka/src/confluent_kafka.o build/temp.linux-x86_64-3.6/confluent_kafka/src/Producer.o build/temp.linux-x86_64-3.6/confluent_kafka/src/Consumer.o build/temp.linux-x86_64-3.6/confluent_kafka/src/Metadata.o build/temp.linux-x86_64-3.6/confluent_kafka/src/AdminTypes.o build/temp.linux-x86_64-3.6/confluent_kafka/src/Admin.o -lrdkafka -o build/lib.linux-x86_64-3.6/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/confluent_kafka
creating build/bdist.linux-x86_64/egg/confluent_kafka/avro
copying build/lib.linux-x86_64-3.6/confluent_kafka/avro/error.py -> build/bdist.linux-x86_64/egg/confluent_kafka/avro
copying build/lib.linux-x86_64-3.6/confluent_kafka/avro/cached_schema_registry_client.py -> build/bdist.linux-x86_64/egg/confluent_kafka/avro
copying build/lib.linux-x86_64-3.6/confluent_kafka/avro/load.py -> build/bdist.linux-x86_64/egg/confluent_kafka/avro
copying build/lib.linux-x86_64-3.6/confluent_kafka/avro/__init__.py -> build/bdist.linux-x86_64/egg/confluent_kafka/avro
creating build/bdist.linux-x86_64/egg/confluent_kafka/avro/serializer
copying build/lib.linux-x86_64-3.6/confluent_kafka/avro/serializer/message_serializer.py -> build/bdist.linux-x86_64/egg/confluent_kafka/avro/serializer
copying build/lib.linux-x86_64-3.6/confluent_kafka/avro/serializer/__init__.py -> build/bdist.linux-x86_64/egg/confluent_kafka/avro/serializer
creating build/bdist.linux-x86_64/egg/confluent_kafka/admin
copying build/lib.linux-x86_64-3.6/confluent_kafka/admin/__init__.py -> build/bdist.linux-x86_64/egg/confluent_kafka/admin
creating build/bdist.linux-x86_64/egg/confluent_kafka/kafkatest
copying build/lib.linux-x86_64-3.6/confluent_kafka/kafkatest/verifiable_producer.py -> build/bdist.linux-x86_64/egg/confluent_kafka/kafkatest
copying build/lib.linux-x86_64-3.6/confluent_kafka/kafkatest/__init__.py -> build/bdist.linux-x86_64/egg/confluent_kafka/kafkatest
copying build/lib.linux-x86_64-3.6/confluent_kafka/kafkatest/verifiable_consumer.py -> build/bdist.linux-x86_64/egg/confluent_kafka/kafkatest
copying build/lib.linux-x86_64-3.6/confluent_kafka/kafkatest/verifiable_client.py -> build/bdist.linux-x86_64/egg/confluent_kafka/kafkatest
copying build/lib.linux-x86_64-3.6/confluent_kafka/__init__.py -> build/bdist.linux-x86_64/egg/confluent_kafka
copying build/lib.linux-x86_64-3.6/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/confluent_kafka
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/avro/error.py to error.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/avro/cached_schema_registry_client.py to cached_schema_registry_client.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/avro/load.py to load.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/avro/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/avro/serializer/message_serializer.py to message_serializer.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/avro/serializer/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/admin/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/kafkatest/verifiable_producer.py to verifiable_producer.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/kafkatest/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/kafkatest/verifiable_consumer.py to verifiable_consumer.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/kafkatest/verifiable_client.py to verifiable_client.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/__init__.py to __init__.cpython-36.pyc
creating stub loader for confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/confluent_kafka/cimpl.py to cimpl.cpython-36.pyc
installing package data to build/bdist.linux-x86_64/egg
running install_data
copying LICENSE.txt -> build/bdist.linux-x86_64/egg/
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying confluent_kafka.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying confluent_kafka.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying confluent_kafka.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying confluent_kafka.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying confluent_kafka.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
confluent_kafka.__pycache__.cimpl.cpython-36: module references __file__
creating 'dist/confluent_kafka-0.11.5-py3.6-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing confluent_kafka-0.11.5-py3.6-linux-x86_64.egg
creating /home/mgutsche/anaconda3/envs/vavel_env/lib/python3.6/site-packages/confluent_kafka-0.11.5-py3.6-linux-x86_64.egg
Extracting confluent_kafka-0.11.5-py3.6-linux-x86_64.egg to /home/mgutsche/anaconda3/envs/vavel_env/lib/python3.6/site-packages
Adding confluent-kafka 0.11.5 to easy-install.pth file

Installed /home/mgutsche/anaconda3/envs/vavel_env/lib/python3.6/site-packages/confluent_kafka-0.11.5-py3.6-linux-x86_64.egg
Processing dependencies for confluent-kafka==0.11.5
Finished processing dependencies for confluent-kafka==0.11.5

From a different directory the same error message appears:

python -c "import confluent_kafka"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/mgutsche/anaconda3/envs/vavel_env/lib/python3.6/site-packages/confluent_kafka-0.11.5-py3.6-linux-x86_64.egg/confluent_kafka/__init__.py", line 2, in <module>
    from .cimpl import (Consumer,  # noqa
ImportError: /home/mgutsche/anaconda3/envs/vavel_env/lib/python3.6/site-packages/confluent_kafka-0.11.5-py3.6-linux-x86_64.egg/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so: undefined symbol: rd_kafka_topic_result_name

So somehow, librdkafka is not linked at all, but the setup.py install seems to be contend with it.

edenhill commented 6 years ago

This is the output of my linux build and ldd, with librdkafka as a linked library:

maglun@eden:~/src/confluent-kafka-python (master)$ python setup.py build
running build
running build_py
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/confluent_kafka
copying confluent_kafka/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka
creating build/lib.linux-x86_64-2.7/confluent_kafka/admin
copying confluent_kafka/admin/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka/admin
creating build/lib.linux-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/load.py -> build/lib.linux-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.linux-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/error.py -> build/lib.linux-x86_64-2.7/confluent_kafka/avro
copying confluent_kafka/avro/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka/avro
creating build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
copying confluent_kafka/kafkatest/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka/kafkatest
creating build/lib.linux-x86_64-2.7/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-2.7/confluent_kafka/avro/serializer
copying confluent_kafka/avro/serializer/__init__.py -> build/lib.linux-x86_64-2.7/confluent_kafka/avro/serializer
running build_ext
building 'confluent_kafka.cimpl' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/confluent_kafka
creating build/temp.linux-x86_64-2.7/confluent_kafka/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-2.7/confluent_kafka/src/confluent_kafka.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c confluent_kafka/src/Producer.c -o build/temp.linux-x86_64-2.7/confluent_kafka/src/Producer.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c confluent_kafka/src/Consumer.c -o build/temp.linux-x86_64-2.7/confluent_kafka/src/Consumer.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c confluent_kafka/src/Metadata.c -o build/temp.linux-x86_64-2.7/confluent_kafka/src/Metadata.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c confluent_kafka/src/AdminTypes.c -o build/temp.linux-x86_64-2.7/confluent_kafka/src/AdminTypes.o
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c confluent_kafka/src/Admin.c -o build/temp.linux-x86_64-2.7/confluent_kafka/src/Admin.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-nbjU53/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/confluent_kafka/src/confluent_kafka.o build/temp.linux-x86_64-2.7/confluent_kafka/src/Producer.o build/temp.linux-x86_64-2.7/confluent_kafka/src/Consumer.o build/temp.linux-x86_64-2.7/confluent_kafka/src/Metadata.o build/temp.linux-x86_64-2.7/confluent_kafka/src/AdminTypes.o build/temp.linux-x86_64-2.7/confluent_kafka/src/Admin.o -lrdkafka -o build/lib.linux-x86_64-2.7/confluent_kafka/cimpl.so
maglun@eden:~/src/confluent-kafka-python (master)$ 
maglun@eden:~/src/confluent-kafka-python (master)$ 
maglun@eden:~/src/confluent-kafka-python (master)$ 
maglun@eden:~/src/confluent-kafka-python (master)$ find . -name '*.so'
./build/lib.linux-x86_64-2.7/confluent_kafka/cimpl.so
maglun@eden:~/src/confluent-kafka-python (master)$ ldd ./build/lib.linux-x86_64-2.7/confluent_kafka/cimpl.so
    linux-vdso.so.1 (0x00007ffe01bc7000)
    librdkafka.so.1 => /usr/local/lib/librdkafka.so.1 (0x00007f984c2c6000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f984bed5000)
    libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f984bcba000)
    libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f984ba50000)
    liblz4.so.1 => /usr/lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f984b834000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f984b496000)
    libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f984b01e000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f984ae01000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f984abfd000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f984a9de000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f984a7d6000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f984c7d1000)
edenhill commented 6 years ago

I'm guessing anaconda is providing some linker flags that makes it ignore librdkafka, maybe as-needed.

mmgutsche commented 6 years ago

You seem to be right, if I call my standard python ldd gets me a little bit further

[mgutsche@sms1.local@DE-LX-3J0545J confluent-kafka-python]$ ldd build/lib.linux-x86_64-3.6/confluent_kafka/cimpl.cpython-36m-x86_64-linux-gnu.so 
    linux-vdso.so.1 (0x00007ffd2b7d7000)
    librdkafka.so.1 => not found
    libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007fa9cdb77000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa9cd959000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fa9cd5a3000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fa9cd39f000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007fa9cd19c000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fa9cce51000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa9ce2f2000)

, but I need to have it work in my anaconda environment anyway. Any idea about any google key words for my issue?

edenhill commented 6 years ago

this seems related: https://groups.google.com/a/continuum.io/forum/#!msg/anaconda/fUpqjkK5jEI/qyoXA01n6nEJ

for the standard build: I suggest you install librdkafka in /usr to avoid having to pass compiler and linker flags around: ./configure --prefix=/usr && make -j && sudo make install

rnpridgeon commented 6 years ago

closing for now due to inactivity. If you have further questions since @edenhill's suggestion please feel free to reopen

JFKenso commented 5 years ago

Hi,

Whilst attempting to build the same library package, specifically confluent_kafka, I'm seeing the same error. The build and deploy runs well, but at the python shell I'm getting the following failure:

Python 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import confluent_kafka Traceback (most recent call last): File "", line 1, in File "/home/pi/.local/lib/python2.7/site-packages/confluent_kafka/init.py", line 2, in from .cimpl import (Consumer, # noqa ImportError: /home/pi/.local/lib/python2.7/site-packages/confluent_kafka/cimpl.so: undefined symbol: rd_kafka_topic_result_name quit()

The ldd for the cimpl.so in question is: ldd /home/pi/.local/lib/python2.7/site-packages/confluent_kafka/cimpl.so linux-vdso.so.1 (0x7edf1000) /usr/lib/arm-linux-gnueabihf/libarmmem.so (0x76ef7000) librdkafka.so.1 => /usr/lib/arm-linux-gnueabihf/librdkafka.so.1 (0x76e53000) libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76e2a000) libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76ceb000) /lib/ld-linux-armhf.so.3 (0x76f36000) libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0x76cc4000) libcrypto.so.1.1 => /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1 (0x76aee000) liblz4.so.1 => /usr/lib/arm-linux-gnueabihf/liblz4.so.1 (0x76acd000) libssl.so.1.1 => /usr/lib/arm-linux-gnueabihf/libssl.so.1.1 (0x76a6c000) libsasl2.so.2 => /usr/lib/arm-linux-gnueabihf/libsasl2.so.2 (0x76a45000) librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x76a2e000) libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0x76a1b000) libresolv.so.2 => /lib/arm-linux-gnueabihf/libresolv.so.2 (0x769f6000)

The environment is: PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)" NAME="Raspbian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" ID=raspbian ID_LIKE=debian

I've received this error both when compiling from source, and embedded whilst installing azure-event-hubs-for-kafka.

Keen for any suggestions or help.

edenhill commented 5 years ago

You will need at least librdkafka v0.11.6, available from confluent: https://docs.confluent.io/current/installation/installing_cp/deb-ubuntu.html#systemd-ubuntu-debian-install

edenhill commented 5 years ago

For Raspian you will probably need to build librdkafka from source

JFKenso commented 5 years ago

Excellent, thank you. I was building from source, however the setup.sh script provided within the Azure repo installed 'librdkafka-dev' via apt-get install. This seemingly caused a version conflict, as this deployed a significantly older version and then tried to deploy the latest librdkafka.

kush-13 commented 5 years ago

You need librdkafka v1.0.0 or later. If it is not available in your distro, uninstall the package and build it from source using the instructions here

For Raspian you will probably need to build librdkafka from source

My OS

pi@raspberrypi:~/Desktop/python-kafka-example $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

I was able to build librdkafka from source

pi@raspberrypi:~/Desktop/librdkafka $ ./configure --install-deps
checking for OS or distribution... ok (raspbian)
checking for C compiler from CC env... failed
checking for gcc (by command)... ok
checking for C++ compiler from CXX env... failed
checking for C++ compiler (g++)... ok
checking executable ld... ok
checking executable nm... ok
checking executable objdump... ok
checking executable strip... ok
checking for pkgconfig (by command)... ok
checking for install (by command)... ok
checking for PIC (by compile)... ok
checking for GNU-compatible linker options... ok
checking for GNU linker-script ld flag... ok
checking for __atomic_32 (by compile)... ok
checking for __atomic_64 (by compile)... failed
checking for __atomic_64_lib (by compile)... ok
checking for socket (by compile)... ok
parsing version '0x010200ff'... ok (1.2.0)
checking for librt (by pkg-config)... failed
checking for librt (by compile)... ok
checking for libpthread (by pkg-config)... failed
checking for libpthread (by compile)... ok
checking for c11threads (by pkg-config)... failed
checking for c11threads (by compile)... ok
checking for libdl (by pkg-config)... failed
checking for libdl (by compile)... ok
checking for zlib (by pkg-config)... ok
checking for libcrypto (by pkg-config)... ok
checking for libssl (by pkg-config)... ok
checking for libsasl2 (by pkg-config)... failed
checking for libsasl2 (by compile)... failed
checking for libsasl (by pkg-config)... failed
checking for libsasl (by compile)... failed (disable)
checking for libzstd (by pkg-config)... failed
checking for libzstd (by compile)... failed
building dependency libzstd... ok (from source)
checking for libzstd (by pkg-config)... failed
checking for libzstd (by compile)... ok
checking for libm (by pkg-config)... failed
checking for libm (by compile)... ok
checking for liblz4 (by pkg-config)... failed
checking for liblz4 (by compile)... failed (disable)
checking for rapidjson (by compile)... failed (disable)
checking for crc32chw (by compile)... failed (disable)
checking for regex (by compile)... ok
checking for strndup (by compile)... ok
checking for strerror_r (by compile)... ok
checking for pthread_setname_gnu (by compile)... ok
checking for nm (by env NM)... ok (cached)
checking for python (by command)... ok
Generated Makefile.config
Generated config.h

Configuration summary:
  prefix                   /usr/local
  MKL_DISTRO               raspbian
  SOLIB_EXT                .so
  ARCH                     armv7l
  CPU                      generic
  GEN_PKG_CONFIG           y
  ENABLE_ZSTD              y
  ENABLE_SSL               y
  ENABLE_GSSAPI            y
  ENABLE_DEVEL             n
  ENABLE_VALGRIND          n
  ENABLE_REFCNT_DEBUG      n
  ENABLE_SHAREDPTR_DEBUG   n
  ENABLE_LZ4_EXT           y
  ENABLE_C11THREADS        y
  MKL_APP_NAME             librdkafka
  MKL_APP_DESC_ONELINE     The Apache Kafka C/C++ library
  CC                       gcc
  CXX                      g++
  LD                       ld
  NM                       nm
  OBJDUMP                  objdump
  STRIP                    strip
  CPPFLAGS                 -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align
  PKG_CONFIG               pkg-config
  INSTALL                  install
  LIB_LDFLAGS              -shared -Wl,-soname,$(LIBFILENAME)
  LDFLAG_LINKERSCRIPT      -Wl,--version-script=
  LDFLAGS                  -Wl,--as-needed
  LIBS                     -lm /home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/lib/libzstd.a -lssl -lcrypto -lz -ldl -lpthread -lrt -latomic -lpthread -lrt -latomic
  RDKAFKA_VERSION_STR      1.2.0
  MKL_APP_VERSION          1.2.0
  CFLAGS                   -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include
  CXXFLAGS                 -Wno-non-virtual-dtor
  SYMDUMPER                $(NM) -D
  exec_prefix              /usr/local
  bindir                   /usr/local/bin
  sbindir                  /usr/local/sbin
  libexecdir               /usr/local/libexec
  datadir                  /usr/local/share
  sysconfdir               /usr/local/etc
  sharedstatedir           /usr/local/com
  localstatedir            /usr/local/var
  libdir                   /usr/local/lib
  includedir               /usr/local/include
  infodir                  /usr/local/info
  mandir                   /usr/local/man
  BUILT_WITH               GCC GXX PKGCONFIG INSTALL GNULD LDS C11THREADS LIBDL PLUGINS ZLIB SSL ZSTD HDRHISTOGRAM SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER
Generated config.cache

Now type 'make' to build
pi@raspberrypi:~/Desktop/librdkafka $ make
make[1]: Entering directory '/home/pi/Desktop/librdkafka/src'
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka.c -o rdkafka.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_broker.c -o rdkafka_broker.o
rdkafka_broker.c: In function ‘rd_kafka_broker_op_serve’:
rdkafka_broker.c:2621:25: warning: ‘strncpy’ output may be truncated copying 255 bytes from a string of length 255 [-Wstringop-truncation]
                         strncpy(rkb->rkb_nodename,
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~
                                 rko->rko_u.node.nodename,
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~
                                 sizeof(rkb->rkb_nodename)-1);
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
rdkafka_broker.c:2662:25: warning: ‘strncpy’ output may be truncated copying 255 bytes from a string of length 255 [-Wstringop-truncation]
                         strncpy(rkb->rkb_name, brokername,
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                 sizeof(rkb->rkb_name)-1);
                                 ~~~~~~~~~~~~~~~~~~~~~~~~
rdkafka_broker.c: In function ‘rd_kafka_broker_update’:
rdkafka_broker.c:5299:25: warning: ‘strncpy’ output may be truncated copying 255 bytes from a string of length 255 [-Wstringop-truncation]
                         strncpy(rko->rko_u.node.nodename, nodename,
                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     sizeof(rko->rko_u.node.nodename)-1);
     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_msg.c -o rdkafka_msg.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_topic.c -o rdkafka_topic.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_conf.c -o rdkafka_conf.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_timer.c -o rdkafka_timer.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_offset.c -o rdkafka_offset.o
In file included from rd.h:68,
                 from rdkafka_int.h:44,
                 from rdkafka_offset.c:51:
rdkafka_offset.c: In function ‘rd_kafka_offset_file_init’:
rdkafka_offset.c:893:51: warning: ‘snprintf’ output may be truncated before the last format character [-Wformat-truncation=]
                 rd_snprintf(spath, sizeof(spath), "%s%s%s",
                                                   ^~~~~~~~
rdposix.h:85:36: note: in definition of macro ‘rd_snprintf’
 #define rd_snprintf(...)  snprintf(__VA_ARGS__)
                                    ^~~~~~~~~~~
rdposix.h:85:27: note: ‘snprintf’ output 1 or more bytes (assuming 4097) into a destination of size 4096
 #define rd_snprintf(...)  snprintf(__VA_ARGS__)
                           ^~~~~~~~~~~~~~~~~~~~~
rdkafka_offset.c:893:17: note: in expansion of macro ‘rd_snprintf’
                 rd_snprintf(spath, sizeof(spath), "%s%s%s",
                 ^~~~~~~~~~~
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_transport.c -o rdkafka_transport.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_buf.c -o rdkafka_buf.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_queue.c -o rdkafka_queue.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_op.c -o rdkafka_op.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_request.c -o rdkafka_request.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_cgrp.c -o rdkafka_cgrp.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_pattern.c -o rdkafka_pattern.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_partition.c -o rdkafka_partition.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_subscription.c -o rdkafka_subscription.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_assignor.c -o rdkafka_assignor.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_range_assignor.c -o rdkafka_range_assignor.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_roundrobin_assignor.c -o rdkafka_roundrobin_assignor.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_feature.c -o rdkafka_feature.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdcrc32.c -o rdcrc32.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c crc32c.c -o crc32c.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdmurmur2.c -o rdmurmur2.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdaddr.c -o rdaddr.o
In file included from rd.h:68,
                 from rdaddr.c:30:
rdaddr.c: In function ‘rd_sockaddr2str’:
rdaddr.c:77:6: warning: ‘%s’ directive output may be truncated writing up to 63 bytes into a region of size between 60 and 61 [-Wformat-truncation=]
      "%s:%s",
      ^~~~~~~
rdaddr.c:79:6:
      portstr);
      ~~~~~~~
rdposix.h:85:36: note: in definition of macro ‘rd_snprintf’
 #define rd_snprintf(...)  snprintf(__VA_ARGS__)
                                    ^~~~~~~~~~~
rdposix.h:85:27: note: ‘snprintf’ output between 2 and 66 bytes into a destination of size 62
 #define rd_snprintf(...)  snprintf(__VA_ARGS__)
                           ^~~~~~~~~~~~~~~~~~~~~
rdaddr.c:76:4: note: in expansion of macro ‘rd_snprintf’
    rd_snprintf(ret[reti]+len, sizeof(ret[reti])-len,
    ^~~~~~~~~~~
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdrand.c -o rdrand.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdlist.c -o rdlist.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c tinycthread.c -o tinycthread.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c tinycthread_extra.c -o tinycthread_extra.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdlog.c -o rdlog.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdstring.c -o rdstring.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_event.c -o rdkafka_event.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_metadata.c -o rdkafka_metadata.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdregex.c -o rdregex.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdports.c -o rdports.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_metadata_cache.c -o rdkafka_metadata_cache.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdavl.c -o rdavl.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_sasl.c -o rdkafka_sasl.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_sasl_plain.c -o rdkafka_sasl_plain.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_interceptor.c -o rdkafka_interceptor.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_msgset_writer.c -o rdkafka_msgset_writer.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_msgset_reader.c -o rdkafka_msgset_reader.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_header.c -o rdkafka_header.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_admin.c -o rdkafka_admin.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_aux.c -o rdkafka_aux.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_background.c -o rdkafka_background.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_idempotence.c -o rdkafka_idempotence.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_cert.c -o rdkafka_cert.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdvarint.c -o rdvarint.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdbuf.c -o rdbuf.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdunittest.c -o rdunittest.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_sasl_scram.c -o rdkafka_sasl_scram.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_sasl_oauthbearer.c -o rdkafka_sasl_oauthbearer.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c snappy.c -o snappy.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdgz.c -o rdgz.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_zstd.c -o rdkafka_zstd.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdhdrhistogram.c -o rdhdrhistogram.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_ssl.c -o rdkafka_ssl.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_lz4.c -o rdkafka_lz4.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -O3 -c xxhash.c -o xxhash.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -O3 -c lz4.c -o lz4.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -O3 -c lz4frame.c -o lz4frame.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -O3 -c lz4hc.c -o lz4hc.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rddl.c -o rddl.o
gcc -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -DLIBRDKAFKA_GIT_VERSION="\"v1.2.0-7-gc3febe\"" -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -c rdkafka_plugin.c -o rdkafka_plugin.o
Generating linker script librdkafka.lds from rdkafka.h
Creating shared library librdkafka.so.1
gcc -Wl,--as-needed -shared -Wl,-soname,librdkafka.so.1 -Wl,--version-script=librdkafka.lds rdkafka.o rdkafka_broker.o rdkafka_msg.o rdkafka_topic.o rdkafka_conf.o rdkafka_timer.o rdkafka_offset.o rdkafka_transport.o rdkafka_buf.o rdkafka_queue.o rdkafka_op.o rdkafka_request.o rdkafka_cgrp.o rdkafka_pattern.o rdkafka_partition.o rdkafka_subscription.o rdkafka_assignor.o rdkafka_range_assignor.o rdkafka_roundrobin_assignor.o rdkafka_feature.o rdcrc32.o crc32c.o rdmurmur2.o rdaddr.o rdrand.o rdlist.o tinycthread.o tinycthread_extra.o rdlog.o rdstring.o rdkafka_event.o rdkafka_metadata.o rdregex.o rdports.o rdkafka_metadata_cache.o rdavl.o rdkafka_sasl.o rdkafka_sasl_plain.o rdkafka_interceptor.o rdkafka_msgset_writer.o rdkafka_msgset_reader.o rdkafka_header.o rdkafka_admin.o rdkafka_aux.o rdkafka_background.o rdkafka_idempotence.o rdkafka_cert.o rdvarint.o rdbuf.o rdunittest.o rdkafka_sasl_scram.o rdkafka_sasl_oauthbearer.o snappy.o rdgz.o rdkafka_zstd.o rdhdrhistogram.o rdkafka_ssl.o rdkafka_lz4.o xxhash.o lz4.o lz4frame.o lz4hc.o rddl.o rdkafka_plugin.o -o librdkafka.so.1 -lm /home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/lib/libzstd.a -lssl -lcrypto -lz -ldl -lpthread -lrt -latomic -lpthread -lrt -latomic
Creating static library librdkafka.a
ar rcs librdkafka.a rdkafka.o rdkafka_broker.o rdkafka_msg.o rdkafka_topic.o rdkafka_conf.o rdkafka_timer.o rdkafka_offset.o rdkafka_transport.o rdkafka_buf.o rdkafka_queue.o rdkafka_op.o rdkafka_request.o rdkafka_cgrp.o rdkafka_pattern.o rdkafka_partition.o rdkafka_subscription.o rdkafka_assignor.o rdkafka_range_assignor.o rdkafka_roundrobin_assignor.o rdkafka_feature.o rdcrc32.o crc32c.o rdmurmur2.o rdaddr.o rdrand.o rdlist.o tinycthread.o tinycthread_extra.o rdlog.o rdstring.o rdkafka_event.o rdkafka_metadata.o rdregex.o rdports.o rdkafka_metadata_cache.o rdavl.o rdkafka_sasl.o rdkafka_sasl_plain.o rdkafka_interceptor.o rdkafka_msgset_writer.o rdkafka_msgset_reader.o rdkafka_header.o rdkafka_admin.o rdkafka_aux.o rdkafka_background.o rdkafka_idempotence.o rdkafka_cert.o rdvarint.o rdbuf.o rdunittest.o rdkafka_sasl_scram.o rdkafka_sasl_oauthbearer.o snappy.o rdgz.o rdkafka_zstd.o rdhdrhistogram.o rdkafka_ssl.o rdkafka_lz4.o xxhash.o lz4.o lz4frame.o lz4hc.o rddl.o rdkafka_plugin.o
Creating librdkafka.so symlink
rm -f "librdkafka.so" && ln -s "librdkafka.so.1" "librdkafka.so"
Generating pkg-config file rdkafka.pc
Generating pkg-config file rdkafka-static.pc
Checking librdkafka integrity
librdkafka.so.1                OK
librdkafka.a                   OK
Symbol visibility              OK
make[1]: Leaving directory '/home/pi/Desktop/librdkafka/src'
make[1]: Entering directory '/home/pi/Desktop/librdkafka/src-cpp'
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c RdKafka.cpp -o RdKafka.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c ConfImpl.cpp -o ConfImpl.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c HandleImpl.cpp -o HandleImpl.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c ConsumerImpl.cpp -o ConsumerImpl.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c ProducerImpl.cpp -o ProducerImpl.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c KafkaConsumerImpl.cpp -o KafkaConsumerImpl.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c TopicImpl.cpp -o TopicImpl.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c TopicPartitionImpl.cpp -o TopicPartitionImpl.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c MessageImpl.cpp -o MessageImpl.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c HeadersImpl.cpp -o HeadersImpl.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c QueueImpl.cpp -o QueueImpl.o
g++ -MD -MP -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -c MetadataImpl.cpp -o MetadataImpl.o
Creating shared library librdkafka++.so.1
gcc -Wl,--as-needed -shared -Wl,-soname,librdkafka++.so.1 RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o -o librdkafka++.so.1 -L../src -lrdkafka -lstdc++
Creating static library librdkafka++.a
ar rcs librdkafka++.a RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o
Creating librdkafka++.so symlink
rm -f "librdkafka++.so" && ln -s "librdkafka++.so.1" "librdkafka++.so"
Generating pkg-config file rdkafka++.pc
Generating pkg-config file rdkafka++-static.pc
Checking librdkafka++ integrity
librdkafka++.so.1              OK
librdkafka++.a                 OK
make[1]: Leaving directory '/home/pi/Desktop/librdkafka/src-cpp'
make -C examples
make[1]: Entering directory '/home/pi/Desktop/librdkafka/examples'
gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -I../src rdkafka_example.c -o rdkafka_example -Wl,--as-needed \
        ../src/librdkafka.a -lm /home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/lib/libzstd.a -lssl -lcrypto -lz -ldl -lpthread -lrt -latomic -lpthread -lrt -latomic
# rdkafka_example is ready
#
# Run producer (write messages on stdin)
./rdkafka_example -P -t <topic> -p <partition>

# or consumer
./rdkafka_example -C -t <topic> -p <partition>

#
# More usage options:
./rdkafka_example -h
gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -I../src rdkafka_performance.c -o rdkafka_performance -Wl,--as-needed \
        ../src/librdkafka.a -lm /home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/lib/libzstd.a -lssl -lcrypto -lz -ldl -lpthread -lrt -latomic -lpthread -lrt -latomic
# rdkafka_performance is ready
#
# Run producer
./rdkafka_performance -P -t <topic> -p <partition> -s <msgsize>

# or consumer
./rdkafka_performance -C -t <topic> -p <partition>

#
# More usage options:
./rdkafka_performance -h
g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp rdkafka_example.cpp -o rdkafka_example_cpp -Wl,--as-needed \
        ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm /home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/lib/libzstd.a -lssl -lcrypto -lz -ldl -lpthread -lrt -latomic -lpthread -lrt -latomic -lstdc++
gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -I../src rdkafka_consumer_example.c -o rdkafka_consumer_example -Wl,--as-needed \
        ../src/librdkafka.a -lm /home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/lib/libzstd.a -lssl -lcrypto -lz -ldl -lpthread -lrt -latomic -lpthread -lrt -latomic
# rdkafka_consumer_example is ready
#
./rdkafka_consumer_example <topic[:part]> <topic2[:part]> ..

#
# More usage options:
./rdkafka_consumer_example -h
g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp rdkafka_consumer_example.cpp -o rdkafka_consumer_example_cpp -Wl,--as-needed \
        ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm /home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/lib/libzstd.a -lssl -lcrypto -lz -ldl -lpthread -lrt -latomic -lpthread -lrt -latomic -lstdc++
g++ -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -Wno-non-virtual-dtor -I../src-cpp kafkatest_verifiable_client.cpp -o kafkatest_verifiable_client -Wl,--as-needed \
        ../src-cpp/librdkafka++.a ../src/librdkafka.a -lm /home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/lib/libzstd.a -lssl -lcrypto -lz -ldl -lpthread -lrt -latomic -lpthread -lrt -latomic -lstdc++
In file included from /usr/include/c++/8/map:60,
                 from kafkatest_verifiable_client.cpp:36:
/usr/include/c++/8/bits/stl_tree.h: In member function ‘std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::iterator std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::_M_emplace_hint_unique(std::_Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::const_iterator, _Args&& ...) [with _Args = {const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&>, std::tuple<>}; _Key = std::__cxx11::basic_string<char>; _Val = std::pair<const std::__cxx11::basic_string<char>, Assignment>; _KeyOfValue = std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, Assignment> >; _Compare = std::less<std::__cxx11::basic_string<char> >; _Alloc = std::allocator<std::pair<const std::__cxx11::basic_string<char>, Assignment> >]’:
/usr/include/c++/8/bits/stl_tree.h:2411:7: note: parameter passing for argument of type ‘std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, Assignment>, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, Assignment> >, std::less<std::__cxx11::basic_string<char> >, std::allocator<std::pair<const std::__cxx11::basic_string<char>, Assignment> > >::const_iterator’ {aka ‘std::_Rb_tree_const_iterator<std::pair<const std::__cxx11::basic_string<char>, Assignment> >’} changed in GCC 7.1
       _Rb_tree<_Key, _Val, _KeyOfValue, _Compare, _Alloc>::
       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/c++/8/map:61,
                 from kafkatest_verifiable_client.cpp:36:
/usr/include/c++/8/bits/stl_map.h: In function ‘void msg_consume(RdKafka::KafkaConsumer*, RdKafka::Message*, void*)’:
/usr/include/c++/8/bits/stl_map.h:518:8: note: parameter passing for argument of type ‘std::_Rb_tree<std::__cxx11::basic_string<char>, std::pair<const std::__cxx11::basic_string<char>, Assignment>, std::_Select1st<std::pair<const std::__cxx11::basic_string<char>, Assignment> >, std::less<std::__cxx11::basic_string<char> >, std::allocator<std::pair<const std::__cxx11::basic_string<char>, Assignment> > >::const_iterator’ {aka ‘std::_Rb_tree_const_iterator<std::pair<const std::__cxx11::basic_string<char>, Assignment> >’} changed in GCC 7.1
    __i = _M_t._M_emplace_hint_unique(__i, std::piecewise_construct,
gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -I../src rdkafka_simple_producer.c -o rdkafka_simple_producer -Wl,--as-needed \
        ../src/librdkafka.a -lm /home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/lib/libzstd.a -lssl -lcrypto -lz -ldl -lpthread -lrt -latomic -lpthread -lrt -latomic
gcc -g -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align -I/home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/include -I../src rdkafka_idempotent_producer.c -o rdkafka_idempotent_producer -Wl,--as-needed \
        ../src/librdkafka.a -lm /home/pi/Desktop/librdkafka/mklove/deps/dest/libzstd/usr/lib/libzstd.a -lssl -lcrypto -lz -ldl -lpthread -lrt -latomic -lpthread -lrt -latomic
make[1]: Leaving directory '/home/pi/Desktop/librdkafka/examples'
Updating
CONFIGURATION.md CONFIGURATION.md.tmp differ: byte 418, line 6
Checking  integrity
CONFIGURATION.md               OK
examples/rdkafka_example       OK
examples/rdkafka_performance   OK
examples/rdkafka_example_cpp   OK
make[1]: Entering directory '/home/pi/Desktop/librdkafka/src'
Checking librdkafka integrity
librdkafka.so.1                OK
librdkafka.a                   OK
Symbol visibility              OK
make[1]: Leaving directory '/home/pi/Desktop/librdkafka/src'
make[1]: Entering directory '/home/pi/Desktop/librdkafka/src-cpp'
Creating shared library librdkafka++.so.1
gcc -Wl,--as-needed -shared -Wl,-soname,librdkafka++.so.1 RdKafka.o ConfImpl.o HandleImpl.o ConsumerImpl.o ProducerImpl.o KafkaConsumerImpl.o TopicImpl.o TopicPartitionImpl.o MessageImpl.o HeadersImpl.o QueueImpl.o MetadataImpl.o -o librdkafka++.so.1 -L../src -lrdkafka -lstdc++
Checking librdkafka++ integrity
librdkafka++.so.1              OK
librdkafka++.a                 OK
make[1]: Leaving directory '/home/pi/Desktop/librdkafka/src-cpp'
pi@raspberrypi:~/Desktop/librdkafka $ sudo make install
make[1]: Entering directory '/home/pi/Desktop/librdkafka/src'
Install librdkafka to /usr/local
install -d $DESTDIR/usr/local/include/librdkafka && \
install -d $DESTDIR/usr/local/lib && \
install rdkafka.h $DESTDIR/usr/local/include/librdkafka && \
install librdkafka.a $DESTDIR/usr/local/lib && \
install librdkafka.so.1 $DESTDIR/usr/local/lib && \
[ -f "rdkafka.pc" ] && ( \
        install -d $DESTDIR/usr/local/lib/pkgconfig && \
        install -m 0644 rdkafka.pc $DESTDIR/usr/local/lib/pkgconfig \
) && \
[ -f "rdkafka-static.pc" ] && ( \
        install -d $DESTDIR/usr/local/lib/pkgconfig && \
        install -m 0644 rdkafka-static.pc $DESTDIR/usr/local/lib/pkgconfig \
) && \
(cd $DESTDIR/usr/local/lib && ln -sf librdkafka.so.1 librdkafka.so)
make[1]: Leaving directory '/home/pi/Desktop/librdkafka/src'
make[1]: Entering directory '/home/pi/Desktop/librdkafka/src-cpp'
Install librdkafka++ to /usr/local
install -d $DESTDIR/usr/local/include/librdkafka && \
install -d $DESTDIR/usr/local/lib && \
install rdkafkacpp.h $DESTDIR/usr/local/include/librdkafka && \
install librdkafka++.a $DESTDIR/usr/local/lib && \
install librdkafka++.so.1 $DESTDIR/usr/local/lib && \
[ -f "rdkafka++.pc" ] && ( \
        install -d $DESTDIR/usr/local/lib/pkgconfig && \
        install -m 0644 rdkafka++.pc $DESTDIR/usr/local/lib/pkgconfig \
) && \
[ -f "rdkafka++-static.pc" ] && ( \
        install -d $DESTDIR/usr/local/lib/pkgconfig && \
        install -m 0644 rdkafka++-static.pc $DESTDIR/usr/local/lib/pkgconfig \
) && \
(cd $DESTDIR/usr/local/lib && ln -sf librdkafka++.so.1 librdkafka++.so)
make[1]: Leaving directory '/home/pi/Desktop/librdkafka/src-cpp'
pi@raspberrypi:~/Desktop/librdkafka $ ls
CMakeLists.txt      config.log.old    debian           lds-gen.py            LICENSE.murmur2  LICENSES.txt         Makefile.config  src
CODE_OF_CONDUCT.md  CONFIGURATION.md  dev-conf.sh      LICENSE               LICENSE.pycrc    LICENSE.tinycthread  mklove           src-cpp
config.cache        configure         Doxyfile         LICENSE.crc32c        LICENSE.queue    LICENSE.wingetopt    packaging        STATISTICS.md
config.h            configure.self    examples         LICENSE.hdrhistogram  LICENSE.regexp   mainpage.doxy        README.md        tests
config.log          CONTRIBUTING.md   INTRODUCTION.md  LICENSE.lz4           LICENSE.snappy   Makefile             README.win32     win32
pi@raspberrypi:~/Desktop/librdkafka $ dpkg librdkakfa -l
dpkg: error: need an action option

Type dpkg --help for help about installing and deinstalling packages [*];
Use 'apt' or 'aptitude' for user-friendly package management;
Type dpkg -Dhelp for a list of dpkg debug flag values;
Type dpkg --force-help for a list of forcing options;
Type dpkg-deb --help for help about manipulating *.deb files;

Options marked [*] produce a lot of output - pipe it through 'less' or 'more' !
pi@raspberrypi:~/Desktop/librdkafka $ dpkg --
dpkg: error: need an action option

Type dpkg --help for help about installing and deinstalling packages [*];
Use 'apt' or 'aptitude' for user-friendly package management;
Type dpkg -Dhelp for a list of dpkg debug flag values;
Type dpkg --force-help for a list of forcing options;
Type dpkg-deb --help for help about manipulating *.deb files;

Options marked [*] produce a lot of output - pipe it through 'less' or 'more' !
pi@raspberrypi:~/Desktop/librdkafka $ dpkg --l
--list       --listfiles  --log=

**Now i guess librdkafka is sucessfully installed:[RAN dpkg --list ] . . .

ii  librdkafka++1:armhf                   0.11.6-1.1                          armhf        library implementing the Apache Kafka protocol (C++ bindings)
ii  librdkafka-dev:armhf                  0.11.6-1.1                          armhf        library implementing the Apache Kafka protocol (development headers)
ii  librdkafka1:armhf                     0.11.6-1.1                          armhf        library implementing the Apache Kafka protocol

. . .

after succesfully installing confluent-kafka using pip3 i'm getting the following error

pi@raspberrypi:~/Desktop/python-kafka-example $ python -c "import confluent_kafka" Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/confluent_kafka/init.py", line 2, in from .cimpl import (Consumer, # noqa ImportError: /usr/local/lib/python2.7/dist-packages/confluent_kafka/cimpl.so: undefined symbol: rd_kafka_fatal_error

Need Help !

edenhill commented 5 years ago

Remove the raspbian librdkafka packages by sudo apt purge librdkafka-dev librdkafka1, then install your own librdkafka:

 $ ./configure --prefix=/usr
 $ make
 $ sudo make install

your python client should now work.

kush-13 commented 5 years ago

./configure --prefix=/usr

Thanks for the help