build librdkafka when installing confluent-kafka

[felixthecat]

currently when i do a pip install confluent-kafka, it fails because i haven't built librdkafka yet. it'd be nice if your setup.py could also build librdkafka to make the installation process easier. there are multiple internal services using our client that's a wrapper around confluent-kafka, and we'd like it if they could just do a simple pip install our-client. this is a little more difficult for us since we then have to automate the build process on our end.

basically i'm hoping you can do something similar to what librabbitmq (https://pypi.python.org/pypi/librabbitmq) is doing.

[ZmeiGorynych]

I run into apparently the same error - can it be fixed manually?

   x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-2.7/confluent_kafka/src/confluent_kafka.o
    In file included from confluent_kafka/src/confluent_kafka.c:17:0:
    confluent_kafka/src/confluent_kafka.h:20:32: fatal error: librdkafka/rdkafka.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

[edenhill]

@ZmeiGorynych Download, build and install librdkafka manually:

curl -L https://github.com/edenhill/librdkafka/archive/v0.9.2-RC1.tar.gz | tar xzf -
cd librdkafka-0.9.2-RC1/
./configure --prefix=/usr
make -j
sudo make install

Then try to build/install confluent-kafka-python again.

[edenhill]

Re issue in general: we'll investigate what bootstrapping librdkafka through setup.py might look like.

[ZmeiGorynych]

That seems to have worked, thanks! The only strangeness is that I had to do sudo pip install confluent-kafka as installing as me gave OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/confluent_kafka-0.9.1.2.dist-info'

[ZmeiGorynych]

Please, what's the equivalent for Windows?

When trying to install on Windows 10 Home, I'm getting the following error install_error.txt

[edenhill]

While the client probably works on Windows, it is currently an unsupported platform, so I really can't help you, but looking at the errors it seems that it can't find the librdkafka headers.

[edenhill]

We're looking into bundling prebuilt librdkafka libraries with the Python client (bdist/wheel) on platforms where dependency management is messy (such as OSX and Windows).

[rmax]

FYI, conda has linux and osx packages for librdkafka: conda install -c conda-forge librdkafka

[linsimiao]

and if you are using mac,and you install a python of other version,you should use "./configure --prefix=/usr/local" instead of "./configure --prefix=/usr"

[mtrienis]

In case folks are interested:

You can use manylinux with auditwheel to build standalone Python wheel that bundles librdkafka (and other c dependencies) inside.

Steps:

1. Build confluent-kafka-python wheel python setup.py bdist_wheel
2. Build and install librdkafka: ./configure && make && make install
3. Repair python wheel auditwheel repair confluent_kafka-x.whl -w dist

For step 2. and step 3. make sure you use the appropriate docker container: e.g. quay.io/pypa/manylinux1_x86_64

[edenhill]

@mtrienis Very cool, I'll look into that!

[edenhill]

This is in the works.

[ghost]

tried to install from wheel with deb9

    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fdebug-prefix-map=/build/python3.5-MLq5fN/python3.5-3.5.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/home/user/Downloads/hivemall/workbench/kafka/python/venv/include/python3.5m -c confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-3.5/confluent_kafka/src/confluent_kafka.o
    In file included from confluent_kafka/src/confluent_kafka.c:17:0:
    confluent_kafka/src/confluent_kafka.h:21:32: fatal error: librdkafka/rdkafka.h: No such file or directory
     #include <librdkafka/rdkafka.h>
                                    ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/home/user/Downloads/hivemall/workbench/kafka/python/venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-kd4t9loi/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-_pmetczv-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/Downloads/hivemall/workbench/kafka/python/venv/include/site/python3.5/confluent-kafka" failed with error code 1 in /tmp/pip-build-kd4t9loi/confluent-kafka/

apt-get is outdated with 0.9.3

How to use the bundled pip/wheel install?

[edenhill]

@ItsmeJulian Try this: https://github.com/confluentinc/confluent-kafka-python/issues/184#issuecomment-370916933

[edenhill]

Binary wheels now (v0.11.4) contain librdkafka

[ZeeD]

Are binary wheels only for mac and linux? is windows a supported platform?

[edenhill]

We just merged experimental windows support. Please try the EXPERIMENTAL packages here and let us know the result: https://ci.appveyor.com/project/ConfluentClientEngineering/confluent-kafka-python/build/0.11.0.176/artifacts

[ZeeD]

Hi I have fetched the artifact (confluent_kafka-0.11.5-cp36-cp36m-win_amd64.whl) friday and used to make some tests with a local kafka. So far all is working

[pauelm]

I am currently running into the same issue on Windows. I notice that the latest pip package at this time is 05-22-2020. Are there currently any plans to fix, and if so, is there any active development getting tracking on this? While there have been newer versions released along the way, I see that this same problem was reported unresolved 4 YEARS ago, so it would be nice to get this fixed and packaged so that it can be loaded on Windows properly.

If not, then I will just move forward with other Kafka solutions.

pip-install-4gju67pi\confluent-kafka\confluent_kafka\src\confluent_kafka.h(22): fatal error C1083: Cannot open include file: 'librdkafka/rdkafka.h': No such file or directory

Additional references to this issue: https://stackoverflow.com/questions/51963121/how-to-install-confluent-kafka-python-on-windows-10

Another solution attempt I tried that was also unsuccessful: pip install --pre --index-url https://test.pypi.org/simple/ confluent_kafka

[edenhill]

@pauelm What Python version are you on?

[pauelm]

I am currently on version 3.8.2

[edenhill]

We're still having problems with the cibuildwheel packaging of confluent-kafka for Py 3.8 on Windows, I believe Py 3.7 should work.

[pauelm]

I dropped down to version 3.6.6 and successfully pipped confluent-kafka 1.4.2. I will give this a shot and report back if I run into any issues. Thanks for your help Magnus!

-Paul

Note: I initially dropped down to 3.7.6, but the following errors were thrown: ERROR: Could not find a version that satisfies the requirement confluent-kafka (from versions: none) ERROR: No matching distribution found for confluent-kafka

[cicipp]

It worked on my computer with Python version 3.7.0 and confluent-kafka version 1.3.0.