SEISO-KAFKA.linux-x86_64.so can not link librdkafka.so.1

[frank-s-liu]

Summary of the issue

SEISO-KAFKA.linux-x86_64.so can not link librdkafka.so.1, but can link librdkafka++.so.1

Expected behavior

zeek can load this kafka-plugin

Steps to reproduce

ubuntu desktop 20.04 librdkafka 1.6.x branch ./configure --enable-sasl make install librdkafka in /usr/local/lib zeek-kafka main branch ./configure --with-librdkafka=/usr/local/lib --zeek-dist=$zeek_root make && make install

ldd SEISO-KAFKA.linux-x86_64.so linux-vdso.so.1 (0x00007fff71fc1000) librdkafka++.so.1 => /usr/local/lib/librdkafka++.so.1 (0x00007f56967a7000) libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f56965b4000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5696599000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f56963a7000) librdkafka.so.1 => not found libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5696258000)

readelf -d SEISO-KAFKA.linux-x86_64.so

Dynamic section at offset 0x20c68 contains 28 entries: Tag Type Name/Value 0x0000000000000001 (NEEDED) Shared library: [librdkafka++.so.1] 0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6] 0x000000000000001d (RUNPATH) Library runpath: [/usr/local/lib] 0x000000000000000c (INIT) 0x11000 0x000000000000000d (FINI) 0x1bd14

root@frank-HP-Notebook:/usr/local/lib# pwd /usr/local/lib root@frank-HP-Notebook:/usr/local/lib# ls -l total 51084 -rw-r--r-- 1 root root 3174658 11月 21 14:58 libparaglob.a -rwxr-xr-x 1 root root 7000146 11月 25 11:08 librdkafka++.a -rwxr-xr-x 1 root root 25705810 11月 25 11:08 librdkafka.a lrwxrwxrwx 1 root root 17 11月 25 11:08 librdkafka++.so -> librdkafka++.so.1 lrwxrwxrwx 1 root root 15 11月 25 11:08 librdkafka.so -> librdkafka.so.1 -rwxr-xr-x 1 root root 2466528 11月 25 11:08 librdkafka++.so.1 -rwxr-xr-x 1 root root 13940528 11月 25 11:08 librdkafka.so.1

I know I can copy librdkafka.so.1 to PATH of /usr/lib to avoid this issue, but I don't know what's the runpath to load librdkafka.so.1 when load this plugin

Where applicable, consider providing a patch that uses the end to end testing environment.

Logs, errors, etc.

...

Your environment

• Version of Zeek
  [frank] 5.0.2
• Version or commit hash of the zeek-kafka package [frank] b632187c7afdc75365a256a7d9bb2e5cc81ab46f
• Operating System and version
• [frank] ubuntu desktop 20.04

[ottobackwards]

We have only tested with libkafka 1.4.4, if you look at our github workflow for ci, can you try it with that and see if it works? If it does then the issue is needing 1.6 support I guess

[frank-s-liu]

same issue building with 1.4.x branch of libkafka. I install librdkafka using follows configurations: ./configure --with-librdkafka=/usr/local/lib --zeek-dist=${zeek_src}

in readme file, it suggest only using ./configure ./configure --with-librdkafka=/usr/local/lib I also test this scenario case, same issue.

[frank-s-liu]

follows is the confige summury: ./configure --with-librdkafka=/usr/local/lib Build Directory : build Zeek Source Directory : -- The C compiler identification is GNU 9.4.0 -- The CXX compiler identification is GNU 9.4.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Performing Test cxx17_already_works -- Performing Test cxx17_already_works - Failed -- Performing Test cxx17_works -- Performing Test cxx17_works - Success -- Found BinPAC: /usr/local/zeek/bin/binpac
-- Found Broker: /usr/local/zeek/lib/libbroker.so
-- Zeek executable : /usr/local/zeek/bin/zeek -- Zeek source : -- Zeek build : -- Zeek install prefix : /usr/local/zeek -- Zeek plugin directory: /usr/local/zeek/lib/zeek/plugins -- Zeek debug mode : true -- Zeek unittest support: no -- Found LibRDKafka: /usr/local/lib/librdkafka++.so
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "1.1.1f")
-- Configuring done -- Generating done CMake Warning: Manually-specified variables were not used by the project:

CAF_ROOT_DIR

[frank-s-liu]

the root cause maybe is here 0x000000000000001d (RUNPATH) Library runpath: [/usr/local/lib] it should be rpath while not runpath.

seems cmake enable this link-flag default SET(CMAKE_EXE_LINKER_FLAGS "-Wl,--enable-new-dtags")

[frank-s-liu]

We have only tested with libkafka 1.4.4, if you look at our github workflow for ci, can you try it with that and see if it works? If it does then the issue is needing 1.6 support I guess

Would you please help to check this issue?

[ottobackwards]

https://github.com/SeisoLLC/zeek-kafka/blob/b632187c7afdc75365a256a7d9bb2e5cc81ab46f/.github/workflows/ci.yml#L22 just an FYI, we test with this setup ( UBUNTU 20.04 ) every time

[ottobackwards]

RPATH and RUNPATH are both "things", RUNPATH is valid to have and recommended ( the default unless you explicitly set it)

[ottobackwards]

What is the exact version of kafka you are using, where did you get it ( link ) What compiler and version are you using? are you using gcc or clang?

[ottobackwards]

And what versions of those?

[ottobackwards]

You can reference this docker file for how we build : https://github.com/SeisoLLC/zeek-kafka/blob/main/docker/containers/zeek/Dockerfile.ubuntu:20.04. Can you try your version of kafka built the way we build it here?

[frank-s-liu]

What is the exact version of kafka you are using, where did you get it ( link ) What compiler and version are you using? are you using gcc or clang?

https://github.com/edenhill/librdkafka.git branch 1.4.x

[frank-s-liu]

What is the exact version of kafka you are using, where did you get it ( link ) What compiler and version are you using? are you using gcc or clang?

The C compiler identification is GNU 9.4.0 -- The CXX compiler identification is GNU 9.4.0

[frank-s-liu]

You can reference this docker file for how we build : https://github.com/SeisoLLC/zeek-kafka/blob/main/docker/containers/zeek/Dockerfile.ubuntu:20.04. Can you try your version of kafka built the way we build it here?

yes I checked this docker file, when installing kafkaka, it also runs command ldconfig,