Kirkstone: OpenMP fails to build with thin LTO enabled.

[jaskij]

When using clang as the default toolchain and thin-lto is enabeld in distro configuration, OpenMP fails to build.

The error given is:

/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/aarch64-my_distro-linux/aarch64-my_distro-linux-ld: /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMSupport.a: error adding symbols: file format not recognized

After unpacking libLLVMSupport.a I found out that the object files are, in fact, LLVM IR:

$ file Atomic.cpp.o
Atomic.cpp.o: LLVM IR bitcode

I'm not actually using any LLVM libs on the target, so for now have disabled LTO for clang as a workaround.

[kraj]

Thanks for the report @jaskij yes I know there are some issues with LTO enabling with meta-clang. I think the linker plugin is not configured correctly when linking with lto objects. It needs to be root caused a bit more.

[jaskij]

@kraj can you direct me on how to dig into it?

[kraj]

@kraj can you direct me on how to dig into it?

you can start by looking at the linking step and extract it out of build and try to do manual link. Then you can add -v option to enable verbosity. This will give more info on what compiler is doing when linking. Perhaps its not calling the linker plugin from right directory in cross-toolchain installation tree or some such.

[jaskij]

Looks like it's picking up the GNU linker.

root@yocto:~/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/build# aarch64-my_distro-linux-ld --version
GNU ld (GNU Binutils) 2.38.20220708
Copyright (C) 2022 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/aarch64-my_distro-linux/aarch64-my_distro-linux-clang++ -v -fPIC -target aarch64-my_distro-linux  -mcpu=cortex-a53 -march=armv8-a+crc+crypto -mlittle-endian -Qunused-arguments -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security  --sysroot=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot  -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0=/usr/src/debug/openmp/14.0.3-r0                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0=/usr/src/debug/openmp/14.0.3-r0                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native=  -fvisibility-inlines-hidden  -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-extra -Wno-pedantic -std=c++14 -DNDEBUG  -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fmacro-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0=/usr/src/debug/openmp/14.0.3-r0                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0=/usr/src/debug/openmp/14.0.3-r0                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot=                      -fdebug-prefix-map=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native=  -Wl,-z,relro,-z,now -shared -Wl,-soname,libomptarget.so -o libomptarget/libomptarget.so libomptarget/src/CMakeFiles/omptarget.dir/api.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/device.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/interface.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/interop.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/omptarget.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/rtl.cpp.o  -Wl,-rpath,"\$ORIGIN:"  /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMSupport.a  -ldl  -Wl,--version-script=/home/jaskij/yocto/build/tmp-glibc/work-shared/llvm-project-source-14.0.3-r0/git/openmp/libomptarget/src/exports  -lrt  -ldl  -lm  /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libz.so  /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMDemangle.a
clang version 14.0.3 (https://github.com/llvm/llvm-project 1f9140064dfbfb0bbda8e51306ea51080b2f7aac)
Target: aarch64-my_distro-linux
Thread model: posix
InstalledDir: /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/aarch64-my_distro-linux
Found candidate GCC installation: /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/aarch64-my_distro-linux/11.3.0
Selected GCC installation: /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/aarch64-my_distro-linux/11.3.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/aarch64-my_distro-linux/aarch64-my_distro-linux-ld" --sysroot=/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot -EL --hash-style=both --eh-frame-hdr -m aarch64linux -shared -o libomptarget/libomptarget.so /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot//usr/lib/crti.o /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/aarch64-my_distro-linux/11.3.0/crtbeginS.o -L/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/../lib/aarch64-my_distro-linux -L/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/aarch64-my_distro-linux/11.3.0 -L/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot//lib -L/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot//usr/lib -O1 --hash-style=gnu --as-needed -z relro -z now -soname libomptarget.so libomptarget/src/CMakeFiles/omptarget.dir/api.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/device.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/interface.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/interop.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/omptarget.cpp.o libomptarget/src/CMakeFiles/omptarget.dir/rtl.cpp.o -rpath "\$ORIGIN:" /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMSupport.a -ldl --version-script=/home/jaskij/yocto/build/tmp-glibc/work-shared/llvm-project-source-14.0.3-r0/git/openmp/libomptarget/src/exports -lrt -ldl -lm /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libz.so /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMDemangle.a -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/aarch64-my_distro-linux/11.3.0/crtendS.o /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot//usr/lib/crtn.o
/home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot-native/usr/bin/aarch64-my_distro-linux/aarch64-my_distro-linux-ld: /home/jaskij/yocto/build/tmp-glibc/work/cortexa53-crypto-my_distro-linux/openmp/14.0.3-r0/recipe-sysroot/usr/lib/libLLVMSupport.a: error adding symbols: file format not recognized
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

[jaskij]

You set me on the right track. Knowing that it's the wrong linker, I tested some stuff and adding -fuse-ld=lld to OpenMP recipe worked.

LDFLAGS:append = "-fuse-ld=lld"

Now, why is it using the GNU linker?

[jaskij]

Sorry for the comment spam, but I seem to have found it - using lld distrowide is hidden behind a separate distro features, ld-is-lld. Which is not documented anywhere - search text for ld-is-lld in the repo only returns the one hit in clang.bbclass.

https://github.com/kraj/meta-clang/blob/299a5fd24b7bee2191127c9fef8bd39d807f3492/classes/clang.bbclass#L73

[kraj]

@jaskij good ! yes ld-is-lld is only available with meta-clang and its disabled by default because we want the layer to be as inert as possible when added and give knobs to users choice to enable more clang'ness :)

You are right about documenting this distro feature. Perhaps its worth adding it to README. Patches welcome!!

[jaskij]

@kraj LDFLAGS:append and documentation both against master and then cherrypick the append to Kirkstone? Or how do you want those patches?

[kraj]

yes

[fcuzzocrea]

With the proposed patch (Which has been picked in the kirkstone branch) openmp fails to build here: https://pastebin.com/m00Recv6

Reverting the commit makes the build pass again. Is there something which I am missing?