Undefined references when building jigsaw-lib on Perlmutter

mcarlson801 commented 4 months ago

Albany's weekly Perlmutter compass build failed this week with the following errors:

/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `aligned_alloc@GLIBC_2.16'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `memcpy@GLIBC_2.14'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `__cxa_thread_atexit_impl@GLIBC_2.18'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `clock_gettime@GLIBC_2.17'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/dev_compass_1.4.0-alpha.2/lib/././libicuuc.so.73: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
gmake[2]: *** [src/CMakeFiles/marche-cmd.dir/build.make:100: src/marche] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:156: src/CMakeFiles/marche-cmd.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `aligned_alloc@GLIBC_2.16'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `memcpy@GLIBC_2.14'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `__cxa_thread_atexit_impl@GLIBC_2.18'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `clock_gettime@GLIBC_2.17'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/dev_compass_1.4.0-alpha.2/lib/././libicuuc.so.73: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
gmake[2]: *** [src/CMakeFiles/tripod-cmd.dir/build.make:100: src/tripod] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:130: src/CMakeFiles/tripod-cmd.dir/all] Error 2
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `aligned_alloc@GLIBC_2.16'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `memcpy@GLIBC_2.14'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `__cxa_thread_atexit_impl@GLIBC_2.18'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /opt/cray/pe/gcc-libs/libstdc++.so.6: undefined reference to `clock_gettime@GLIBC_2.17'
/pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/compass_bootstrap/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /pscratch/sd/m/mcarlson/biweeklyCDashPerlmutter-serial/build/compass/envs/dev_compass_1.4.0-alpha.2/lib/././libicuuc.so.73: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
collect2: error: ld returned 1 exit status
gmake[2]: *** [src/CMakeFiles/jigsaw-cmd.dir/build.make:100: src/jigsaw] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:104: src/CMakeFiles/jigsaw-cmd.dir/all] Error 2

Any idea what might be causing this? It looks like there was maybe a similar issue back in January.

xylar commented 4 months ago

@mcarlson801, thanks for reporting this.

I've seen something similar on the OpenMPI feedstock on conda-forge. Based on this comment: https://github.com/conda-forge/openmpi-feedstock/issues/143#issuecomment-2040463544 we might need to install sysroot_linux-64=2.17 explicitly or based on later comments, we might need to make sure $LDFLAGS are getting passed to the linker in the JIGSAW build. I probably won't have time to look into this further until Friday at the earliest.

minrk commented 3 months ago

Since that issue got pretty long with digressions into my own cmake confusion, I'm going to try to make a concise summary:

openmpi is compiled with 2.17 and requires glibc >= 2.17. This runtime requirement is enforced properly.
To link openmpi, you must either have sysroot_linux >=2.17 or add -Wl,--allow-shlib-undefined to your link flags (usually $LDFLAGS)
The gcc package will get sysroot_linux 2.12 by default because newer versions are "weighed down" for maximum compatibility by default (the default will be 2.17 later this summer, I think)
The c-compiler package (gcc_linux-64 aliased as c-compiler or {{ compiler("c"} }} in a recipe) sets -Wl,--allow-shlib-undefined in $LDFLAGS on activation, so most build systems will work by default. If this isn't happening for you, the main reasons would be:
- $LDFLAGS isn't accepted by your build system (it's usually a good idea, I think), or
- gcc is installed instead of c-compiler, so $LDFLAGS is not set. Whether this is right will depend on your circumstances.

So the simple version is to require sysroot_linux>=2.17 in your environment, because then there will be no undefined symbols. The alternative is to leave c_stdlib_version alone and make sure that either $LDFLAGS are passed through, or set -Wl,--allow-shlib-undefined explicitly.

This issue is not specific to openmpi, but actually most libraries built with newer-than-2.12 will encounter some version of this when downstream packages attempt to link them, either in runtime user environments or package building.