Empty profiles when building with gcc/g++

[kramermt]

Issue: Even after adding -ldl to the kmeans Makefile, the resulting profile.coz contained only samples, with no progress points.

The executable still isn't linking/requiring libdl.so:

g++  -o kmeans obj/kmeans-pthread.o -lpthread -ldl
osboxes@osboxes:~/coz/benchmarks/kmeans$ ldd kmeans
    linux-vdso.so.1 (0x00007fff8012e000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6ff775c000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6ff756a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f6ff7798000)

Cause: Turns out gcc, at least on Ubuntu 14.04 and 20.04, is configured to call ld with --as-needed by default, and does so before all command-line-specified specified libraries. So -ldl is effectively --as-needed, and the weak symbol does count as a "need".

clang, on the same system, does not suffer from this, as it excludes the command-line-specified libraries from --as-needed.

Fix: I resolved this locally by changing the Makefile to:

LIBS     := -lpthread -Wl,--no-as-needed -ldl

Which adds not only libdl.so, but several other unneeded libraries to the load list, since the setting then applies to all other libraries gcc adds to the LIBS list.

g++  -o kmeans obj/kmeans-pthread.o -lpthread -Wl,--no-as-needed -ldl
osboxes@osboxes:~/coz/benchmarks/kmeans$ ldd kmeans
    linux-vdso.so.1 (0x00007fffd3feb000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f65e27c9000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f65e27c3000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f65e25e2000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f65e2493000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f65e2478000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f65e2286000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f65e2805000)

Ultimately, the following seems to fix it for gcc, without any adverse side-effects for clang (i.e., clang-linked executables still have the same number of libs in the load list as before this change):

LIBS     := -lpthread -Wl,--push-state,--no-as-needed -ldl -Wl,--pop-state

g++  -o kmeans obj/kmeans-pthread.o -lpthread -Wl,--push-state,--no-as-needed -ldl -Wl,--pop-state
osboxes@osboxes:~/coz/benchmarks/kmeans$ ldd kmeans
    linux-vdso.so.1 (0x00007ffe59be1000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa6ede3c000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa6ede36000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa6edc44000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa6ede78000)

And my profiles now contain progress points. Woot.

Questions:

1. Any idea why more people haven't run into this? Is everyone really using clang? Or distros that don't build gcc with --as-needed in its default link script?
2. Would it be worth adding the --no-as-needed requirement to the README?
3. Should -Wl,--push-state,--no-as-needed -ldl -Wl,--pop-state be added to the benchmark Makefiles?
4. For that matter, why isn't -ldl specified in the benchmark Makefiles as it is?

Happy to make a Pull Request with any updates you'd find worthwhile, @emeryberger or @ccurtsinger .

[emeryberger]

No idea why this hasn't shown up before; thanks a ton for the detective work! I'd gladly take another PR to deal with this as you propose (since the -ldl thing is probably not enough).

[emeryberger]

(Appears to be solved now; thanks!)