libclang.so.13.0.1 can not find libc.so.6 in SDK

[pberginkonsult]

Describe the bug When adding libclang to SDK and used in a build the dynamic linker will use libc.so.6 from the system instead of the one from the SDK. There can be a version

This is found when trying to build rust code with librocksdb-sys using bindgen. Error from rust:

...
     Running `rustc --crate-name build_script_build --edition=2018 /work/sdk/poky-x86/sysroots/core2-64-poky-linux/home/cargo/registry/src/github.com-1ecc6299db9ec823/librocksdb-sys-6.20.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off --cfg 'feature="default"' --cfg 'feature="static"' -C metadata=7c8525cf0d81b61c -C extra-filename=-7c8525cf0d81b61c --out-dir /work/tmp/rust-hello/target/release/build/librocksdb-sys-7c8525cf0d81b61c -L dependency=/work/tmp/rust-hello/target/release/deps --extern bindgen=/work/tmp/rust-hello/target/release/deps/libbindgen-466101c122bb103a.rlib --extern cc=/work/tmp/rust-hello/target/release/deps/libcc-17b88774702fcf3c.rlib --extern glob=/work/tmp/rust-hello/target/release/deps/libglob-b01e7b2cb01490de.rlib --cap-lints allow`
     Running `/work/tmp/rust-hello/target/release/build/librocksdb-sys-7c8525cf0d81b61c/build-script-build`
error: failed to run custom build command for `librocksdb-sys v6.20.3`

Caused by:
  process didn't exit successfully: `/work/tmp/rust-hello/target/release/build/librocksdb-sys-7c8525cf0d81b61c/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at 'Unable to find libclang: "the `libclang` shared library at /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1 could not be opened: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1)"', /work/sdk/poky-x86/sysroots/core2-64-poky-linux/home/cargo/registry/src/github.com-1ecc6299db9ec823/bindgen-0.59.1/src/lib.rs:2117:31
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

When running ldd on libclang.so.13.0.1 and checking versions of libc.so.6:

$ ldd /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1
/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1)
/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/libLLVM-13.so)
/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/libstdc++.so.6)
/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/../lib/libxml2.so.2)
    linux-vdso.so.1 (0x00007ffdf3d47000)
    libLLVM-13.so => /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/libLLVM-13.so (0x00007f1c7dabf000)
    libstdc++.so.6 => /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/libstdc++.so.6 (0x00007f1c7d8b0000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1c7d74f000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1c7d734000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1c7d548000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1c84f0d000)
    libffi.so.8 => /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/../lib/libffi.so.8 (0x00007f1c7d53a000)
    libedit.so.0 => /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/../lib/libedit.so.0 (0x00007f1c7d4ff000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1c7d4f4000)
    libz.so.1 => /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/../lib/libz.so.1 (0x00007f1c7d4da000)
    libxml2.so.2 => /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/../lib/libxml2.so.2 (0x00007f1c7d378000)
    libncurses.so.5 => /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/../lib/libncurses.so.5 (0x00007f1c7d34f000)
    libtinfo.so.5 => /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/../lib/libtinfo.so.5 (0x00007f1c7d320000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1c7d2fe000)
$
$ /lib/x86_64-linux-gnu/libc.so.6 --version | head -n 1
GNU C Library (Ubuntu GLIBC 2.33-0ubuntu5) release release version 2.33.
$ /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/lib/libc.so.6 --version | head -n 1
GNU C Library (GNU libc) stable release version 2.34.

To Reproduce

1. Setup standard Poky build with meta-clang. Used revisions:

   Build Configuration:
   BB_VERSION           = "1.53.0"
   BUILD_SYS            = "x86_64-linux"
   NATIVELSBSTRING      = "universal"
   TARGET_SYS           = "x86_64-poky-linux"
   MACHINE              = "qemux86-64"
   DISTRO               = "poky"
   DISTRO_VERSION       = "3.4+snapshot-d233a42bf3067528be18334924a4b82d8363bcf2"
   TUNE_FEATURES        = "m64 core2"
   TARGET_FPU           = ""
   meta                 
   meta-poky            
   meta-yocto-bsp       = "master-next:d233a42bf3067528be18334924a4b82d8363bcf2"
   meta-clang           = "master:81dca6d47885a35ffe9c05827e83670d500b7535"

2. Add rust and clang to SDK by adding this to the configuration:

   TOOLCHAIN_HOST_TASK:append = " packagegroup-rust-cross-canadian-${MACHINE}"
   CLANGSDK = "1"

3. Build SDK bitbake core-image-minial -cpopulate_sdk
4. Install SDK
5. Initialize SDK and create a minimal rust reproducer

   
   $ . /work/sdk/poky-x86/environment-setup-core2-64-poky-linux
   $ cargo new rust-hello-2
   $ cd rust-hello-2
   $ echo 'librocksdb-sys = "6.20.3"' >> Cargo.toml
   $ cat Cargo.toml 
   [package]
   name = "rust-hello-2"
   version = "0.1.0"
   edition = "2021"

See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies] librocksdb-sys = "6.20.3" $ $ cargo build -v --target x86_64-poky-linux

[pberginkonsult]

This is probably not a meta-clang issue alone. I apologize but forgot one step in the reproducer above which is a patch to relocate ncurses libs in the SDK from /lib to /usr/lib.

$ git diff meta/recipes-core/ncurses/ncurses.inc
diff --git a/meta/recipes-core/ncurses/ncurses.inc b/meta/recipes-core/ncurses/ncurses.inc
index a0ecd8a80b..4dc02652bf 100644
--- a/meta/recipes-core/ncurses/ncurses.inc
+++ b/meta/recipes-core/ncurses/ncurses.inc
@@ -34,6 +34,7 @@ BUILD_CPPFLAGS += "-D_GNU_SOURCE"

 # natives don't generally look in base_libdir
 base_libdir:class-native = "${libdir}"
+base_libdir:class-nativesdk = "${libdir}"

 # Display corruption occurs on 64 bit hosts without these settings
 # This was derrived from the upstream debian ncurses which uses

There are some issue with the paths that the linker searches for libraries. Looking in to libclang.so.13.0.1 issue here I have, with help from this thread https://lists.openembedded.org/g/openembedded-core/topic/80874524#148546 , extracted a lot more information with LD_DEBUG=all. Interesting info related to libclang is this:

       20319:   file=librt.so.1 [0];  needed by /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/libLLVM-13.so [0]
       20319:   find library=librt.so.1 [0]; searching
       20319:    search path=/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/..:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/../lib      (RPATH from file /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/libLLVM-13.so)
       20319:     trying file=/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/../librt.so.1
       20319:     trying file=/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/../lib/librt.so.1
       20319:    search path=/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/..:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib        (RPATH from file /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1)
       20319:     trying file=/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../librt.so.1
       20319:     trying file=/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/librt.so.1
       20319:    search path=/work/tmp/rust-hello/target/debug/deps:/work/tmp/rust-hello/target/debug       (LD_LIBRARY_PATH)
       20319:     trying file=/work/tmp/rust-hello/target/debug/deps/librt.so.1
       20319:     trying file=/work/tmp/rust-hello/target/debug/librt.so.1
       20319:    search cache=/etc/ld.so.cache
       20319:     trying file=/lib/x86_64-linux-gnu/librt.so.1

In the SDK most of the libraries are located in /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib but also some of them, for example librt.so.1, are located in /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/lib. The RPATH of libclang.so.13.0.1 does not include <sysroot>/lib.

$ /work/yocto/poky/build/tmp/sysroots-uninative/x86_64-linux/usr/bin/patchelf-uninative /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1 --print-rpath
$ORIGIN/..:$ORIGIN/../lib

The same seems true for libLLVM (not from this layer):

$ /work/yocto/poky/build/tmp/sysroots-uninative/x86_64-linux/usr/bin/patchelf-uninative /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libLLVM-13.so --print-rpath
$ORIGIN/..:$ORIGIN/../lib

Should $ORIGIN/../..:$ORIGIN/../../lib also be added to RPATH?

But there are also more interesting/strange things in the log from LD_DEBUG=all it reports different RPATH from libclang.so.13.0.1 in different occasions:

<snip>
       20319:    search path=/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../glibc-hwcaps/x86-64-v4:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../glibc-hwcaps/x86-64-v3:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../glibc-hwcaps/x86-64-v2:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../tls/haswell/avx512_1/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../tls/haswell/avx512_1:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../tls/haswell/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../tls/haswell:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../tls/avx512_1/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../tls/avx512_1:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../tls/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../tls:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../haswell/avx512_1/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../haswell/avx512_1:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../haswell/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../haswell:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../avx512_1/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../avx512_1:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/..:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/glibc-hwcaps/x86-64-v4:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/glibc-hwcaps/x86-64-v3:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/glibc-hwcaps/x86-64-v2:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/tls/haswell/avx512_1/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/tls/haswell/avx512_1:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/tls/haswell/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/tls/haswell:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/tls/avx512_1/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/tls/avx512_1:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/tls/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/tls:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/haswell/avx512_1/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/haswell/avx512_1:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/haswell/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/haswell:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/avx512_1/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/avx512_1:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib/x86_64:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib      (RPATH from file /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1)
<snip>
       20319:    search path=/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/..:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib    (RPATH from file /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1)
<snip>
       20319:    search path=/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/..:/work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/../lib    (RPATH from file /work/sdk/poky-x86/sysroots/x86_64-pokysdk-linux/usr/lib/libclang.so.13.0.1)
<snip>

Are there is some other function tampering with RPATH when using Yocto SDK?

[kraj]

Can you try out https://github.com/kraj/meta-clang/pull/533

[pberginkonsult]

Can you try out #533

I have tried #533 and can not see any difference in the dependencies. Have verified that CMAKE_BUILD_WITH_INSTALL_RPATH=ON is passed in do_configure of libcxx. But I can not find any dependency between nativesdk-clang (providing libclnag.so.13.0.1) and libcxx. Should there be one?

[kraj]

@pberginkonsult it seems specific to rust here. Can you try a simple c/c++ example using CLANGCC and see if that works ok ? if that works then we have the issue in llvm bundled with rust