search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
194 stars 3.14k forks source link

Cannot build librdkafka on OracleLinux or AmazonLinux #4467

Open mkmoisen opened 11 months ago

mkmoisen commented 11 months ago

Description

Building librdkafka from source on either OracleLinux or AmazonLinux is failing.

It works fine on RHEL (UBI) but not on the derivative distributions OracleLinux or AmazonLinux.

How to reproduce

Using the following Dockerfile:

#FROM redhat/ubi9:9.2
#FROM oraclelinux:9
FROM amazonlinux:2023

USER 0

ENV APP_ROOT /opt/app-root

WORKDIR $APP_ROOT

RUN yum install wget -y
RUN yum install gcc gcc-c++ make -y
RUN yum install glibc-devel -y
RUN yum install unzip -y

################################################################################
## Install Kafka Confluent
ARG DOCKER_LIBRDKAFKA_VERSION="2.2.0"

RUN yum install cyrus-sasl-gssapi cyrus-sasl-devel -y
RUN yum install -y krb5-workstation
RUN wget https://github.com/confluentinc/librdkafka/archive/refs/tags/v$DOCKER_LIBRDKAFKA_VERSION.zip
RUN unzip v$DOCKER_LIBRDKAFKA_VERSION.zip
RUN cd librdkafka-$DOCKER_LIBRDKAFKA_VERSION/ && ./configure && make && make install

Build the container via:

    docker build -t librdkafkatest -f Dockerfile .

The make install command works for redhat/ubi9:9.2.

However, it fails for both oraclelinux:9 and amazonlinux:2023 with a similar error: d: command not found

amazonlinux:2023

17.01 Install librdkafka to /usr/local
17.01 d $DESTDIR/usr/local/include/librdkafka
17.01 /bin/sh: line 1: d: command not found
17.01 make[1]: [../mklove/Makefile.base:269: lib-install] Error 127 (ignored)
17.01 d $DESTDIR/usr/local/lib
17.01 /bin/sh: line 1: d: command not found
17.01 make[1]: [../mklove/Makefile.base:270: lib-install] Error 127 (ignored)
17.01 rdkafka.h rdkafka_mock.h $DESTDIR/usr/local/include/librdkafka
17.01 /bin/sh: line 1: rdkafka.h: command not found
17.01 make[1]: *** [../mklove/Makefile.base:271: lib-install] Error 127
17.01 make[1]: Leaving directory '/opt/app-root/librdkafka-2.2.0/src'
17.01 make: *** [Makefile:44: install-subdirs] Error 2

oraclelinux:9

17.89 make[1]: Leaving directory '/opt/app-root/librdkafka-2.2.0/src-cpp'
17.90 make[1]: Entering directory '/opt/app-root/librdkafka-2.2.0/src'
17.92 Install librdkafka to /usr/local
17.92 d $DESTDIR/usr/local/include/librdkafka
17.92 /bin/sh: line 1: d: command not found
17.92 make[1]: [../mklove/Makefile.base:269: lib-install] Error 127 (ignored)
17.92 d $DESTDIR/usr/local/lib
17.93 /bin/sh: line 1: d: command not found
17.93 rdkafka.h rdkafka_mock.h $DESTDIR/usr/local/include/librdkafka
17.93 make[1]: [../mklove/Makefile.base:270: lib-install] Error 127 (ignored)
17.93 /bin/sh: line 1: rdkafka.h: command not found
17.93 make[1]: *** [../mklove/Makefile.base:271: lib-install] Error 127
17.93 make[1]: Leaving directory '/opt/app-root/librdkafka-2.2.0/src'
17.93 make: *** [Makefile:44: install-subdirs] Error 2

Using ./configure --instal-deps doesn't help.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

shaaslam commented 11 months ago

I am getting the same error on amazonlinux:2

sh-4.2# make install
Install librdkafka to /usr/local
d $DESTDIR/usr/local/include/librdkafka
/bin/sh: d: command not found
make: [lib-install] Error 127 (ignored)
d $DESTDIR/usr/local/lib
/bin/sh: d: command not found
make: [lib-install] Error 127 (ignored)
rdkafka.h rdkafka_mock.h $DESTDIR/usr/local/include/librdkafka
/bin/sh: rdkafka.h: command not found
make: *** [lib-install] Error 127
navedgaras commented 10 months ago

I have the same issue, too. Are there any workarounds that anyone is aware of?

shaaslam commented 10 months ago

@edgga & @mkmoisen , I did a workaround by using the amd64 amazon image on my local. In Dockerfile replace line no. 1 with FROM amd64/amazonlinux:2

This will download the x86_64 image. If you are working on Silicon Mac, change your docker settings to use rosetta.

mkmoisen commented 10 months ago

@shaaslam

Do you know what is the difference between amd64/amazonlinux:2 and amazonlinux:2 ?

shaaslam commented 10 months ago

amd64/amazonlinux:2 image is for x86_64 platform amazonlinux:2 image is for aarch64

lazamar commented 9 months ago

make is not creating the command correctly. For me which was missing. After I installed it things started working again.

nhaq-confluent commented 7 months ago

@mkmoisen Given the discussion above, can we close this issue?

du-do commented 3 months ago

https://github.com/confluentinc/librdkafka/issues/3974

initanmol commented 2 months ago

On amazonlinux2, I did the following steps to install the current latest version of librdkafka, which was required to install confluent-kafka python package. Try these:

yum update -y
yum groupinstall -y "Development Tools"
yum install -y wget tar gzip openssl-devel zlib-devel libcurl-devel

if cmake --version; then yum remove -y cmake; fi

mkdir /tmp/cmake-install
cd /tmp/cmake-install
wget https://cmake.org/files/v3.25/cmake-3.25.2-linux-aarch64.tar.gz
tar -zxvf cmake-3.25.2-linux-aarch64.tar.gz
cp -r cmake-3.25.2-linux-aarch64/* /usr/local/
ln -s /usr/local/bin/cmake /usr/bin/cmake
if ! cmake --version | grep -i "3.25.2"; then exit 1; fi
cd -

mkdir /tmp/librdkafka-install
cd /tmp/librdkafka-install
wget https://github.com/edenhill/librdkafka/archive/refs/tags/v2.5.0.tar.gz
tar -xvf v2.5.0.tar.gz
cd librdkafka-2.5.0
mkdir build
cd build
cmake ..
make
make install
echo "/usr/local/lib" >> /etc/ld.so.conf
echo "/usr/local/lib64" >> /etc/ld.so.conf
ldconfig

yum install python3-pip python3-devel -y
pip3 install confluent-kafka==2.5.0

python3 -c "import confluent_kafka"