[Bug]: libunwind seems to check pkg-config for paths even when using "bundled"

[bradcray]

Summary of Problem

Description:

On gitter, @stefantalpalaru reported that upon updating to Chapel 2.1, he was seeing warnings from the Chapel linker due to finding incompatible libraries in his /usr/lib library, when /usr/lib64 should've been used (and was—the compile worked correctly other than generating the extraneous warnings). Digging into the source of the issue, he managed to track it down as being due to a problem in util/chplenv/chpl_unwind.py in which we seem to be querying pkg-config to determine the -L path to use for the libunwind library even though CHPL_UNWIND was set to bundled for his build:

Because he had 32-bit versions of libunwind installed on his system in /usr/lib, it seems that this line was adding a -L/usr/lib to his linker line—along with additional -L/usr/lib64 lines, generating the warnings. He also confirmed that this line was in his lib/.../list-libraries file, which is where I believe we collect linker options for third-party packages during the build step.

These scripts haven't changed in this respect for some time, so the hypothesis is that he installed the 32-bit system version of libunwind between his install of 2.0.1 and 2.1.0. Re-building 2.0.1, he saw the same issue there, reassuring us that it isn't a new bug that crept into version 2.1.

Is this issue currently blocking your progress?

Sounds like "no" since programs still compile and run properly, just with extra warnings. We've also discussed rebuilding with CHPL_UNWIND=system or none, imagining that might avoid this bad line.

Configuration Information

See https://matrix.to/#/!PBYDSerrfYujeStENM:gitter.im/$7lZIypyF1gqUzjGl--fGWwSgRirfLtyZj2-wk7DGxIM?via=gitter.im&via=matrix.org&via=envs.net

[mppf]

@stefantalpalaru are you able to share the /usr/lib/chapel/2.1/third-party/libunwind/install/linux64-x86_64-native-llvm-none/lib/pkgconfig/libunwind.pc from the problematic configuration? Thanks

[stefantalpalaru]

$ cat /usr/lib/chapel/2.1/third-party/libunwind/install/linux64-x86_64-native-llvm-none/lib/pkgconfig/libunwind.pc
prefix=/var/tmp/portage/dev-lang/chapel-2.1.0/work/chapel-2.1.0/third-party/libunwind/install/linux64-x86_64-native-llvm-none
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: libunwind
Description: libunwind base library
Version: 1.5.0
Libs: -L${libdir} -lunwind
Libs.private: -llzma -lz
Cflags: -I${includedir}

So that's not the one, because "prefix" doesn't get set to "/usr". I have a better candidate for you:

$ grep '^dependency_libs' /usr/lib/chapel/2.1/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-system/lib/libqthread.la
dependency_libs=' -L/usr/lib -lm'

And that is processed here: https://github.com/chapel-lang/chapel/blob/d8a66b4f8bf84901162c359a950623f91cc37e7c/util/chplenv/third_party_utils.py#L50-L59

which is called from: https://github.com/chapel-lang/chapel/blob/d8a66b4f8bf84901162c359a950623f91cc37e7c/util/chplenv/third_party_utils.py#L290

which is called from: https://github.com/chapel-lang/chapel/blob/d8a66b4f8bf84901162c359a950623f91cc37e7c/util/chplenv/chpl_qthreads.py#L26-L27

If I delete that "dependency_libs" line from "libqthread.la", the error messages go away.

[mppf]

thanks @stefantalpalaru . Can you share /usr/lib/chapel/2.1/third-party/qthread/install/linux64-x86_64-native-llvm-none-flat-jemalloc-system/lib/libqthread.la ? Also can you share the configure log from doing the qthreads build? You might be able to cd third-party/qthread; make clobber; cd ../..; make.

I suspect qthreads is picking up a dependency on your system only available as a 32-bit library.

[stefantalpalaru]

Full build log: https://gist.githubusercontent.com/stefantalpalaru/78d63274aed0229d7e0507e13f8efc97/raw/ccd2b29db0082f23d8dee7ec8ee9202149d49818/chapel-build.log

Installed "libqthread.la": https://gist.github.com/stefantalpalaru/f3fe0ed13127745cae6c91b723a53116

I suspect qthreads is picking up a dependency on your system only available as a 32-bit library.

Everything in my "/usr/lib" is 32-bit. 64-bit libraries are in "/usr/lib64" on Gentoo.

[bradcray]

@mppf: Making sure I'm following: Last night, I was thinking that calling pkg-config for a bundled build was the likely source of the error because it seemed odd we'd use pkg-config for a package that we had built and controlled the location of. But from your comments here, I take it that using pkg-config is expected/by design for even a bundled libunwind?

[mppf]

@bradcray - yes, we want to use .pc (pkg-config files) and/or .la files (libtool files) even for bundled builds because we need to discover the system dependencies that these have.

In further investigation it turns out that the /usr/lib here is coming from qthreads (and ending up in its .la file). The open question is why that is happening. Unfortunately it's not clear to me from the logs.

[bradcray]

Great, thanks for that information, and sorry for heading us down the wrong path with this issue title/description due to not understanding (remembering?) that.

[mppf]

I was able to get a gentoo apptainer image going. Despite my complete inexperience with gentoo, I managed to get the right dependencies installed to build Chapel. I was able to build the default mode without issue with CHPL_LLVM=none. Next I will try with LLVM.

@stefantalpalaru you are using LLVM 18 but I see that sys-devel/clang:18 is a masked package. I'll try with LLVM 17 but I suppose that could be related.

[stefantalpalaru]

It's easier to work with the package manager on this one. You want a Gentoo ~amd64 system (the tilde marks the "unstable" flavour of this rolling-release distro) and my overlay: https://github.com/stefantalpalaru/gentoo-overlay

After that, you can emerge chapel. To see it fail in make check, do USE="abi_x86_32" emerge libunwind first.

My ebuild (package building script) is here: https://github.com/stefantalpalaru/gentoo-overlay/tree/master/dev-lang/chapel

[mppf]

I haven't been able to reproduce even with your overlay. (I am not trying emerge chapel though because I want to be able to edit scripts etc etc).

USE="abi_x86_32" emerge libunwind seems to install libunwind in both /usr/lib and /usr/lib64. On your system, is it only installed in /usr/lib? What about the math library? I think we need to figure out which dependency qthreads is trying to get from /usr/lib and why it is looking there in order to make any progress on this issue.

[stefantalpalaru]

I think it's libtool that puts this in the "configure" output:

LDFLAGS='-L/usr/lib '
LIBS='-lm '

[stefantalpalaru]

With CHPL_HWLOC=system, Chapel's build system passes "--with-hwloc=/usr" to "qthread-src/configure": https://github.com/chapel-lang/chapel/blob/59f311521884dcbf27b8be31f6b5de160612bab0/third-party/qthread/Makefile#L36

This ends up as LDFLAGS="-L/usr/lib $LDFLAGS"] in here: https://github.com/chapel-lang/chapel/blob/59f311521884dcbf27b8be31f6b5de160612bab0/third-party/qthread/qthread-src/config/qthread_check_hwloc.m4#L35

---

That said, I don't think you ever want system library paths appearing in your linker args, while linking Chapel programs, so they should all be removed in handle_la() from "third_party_utils.py".

Right now, you risk linking system libraries when you were asked to use the bundled ones.

[mppf]

hi @stefantalpalaru - thanks for tracking it down.

Now I am able to reproduce (I had to emerge sys-apps/hwloc which was somehow missing until now; then when I use export CHPL_HWLOC=system the qthreads build does include /usr/lib in the .la file).

I've created PR #25405 to address the issue as you have suggested. That will need testing on a few more platforms before merging.

[bradcray]

Thanks so much for looking into this on our side, @mppf, and for reporting the issue and doing so much legwork to determine what was going wrong, @stefantalpalaru!