mpifort 5.0.1 fails with undefined refs to memcpy@GLIBC_2.14 and clock_gettime@GLIBC_2.17

[charlesgwaldman]

Solution to issue cannot be found in the documentation.

• [X] I checked the documentation.

Issue

I am using cmake, gfortran, and openmpi from conda-forge to compile a Fortran package. With cmake 3.28.3, gfortran 13.2.0 and openmpi 5.0.0 everything works. When openmpi upgraded to the latest 5.0.1 version I started getting this error:

-- Check for working Fortran compiler: /home/cgw/miniforge3/envs/mfix-git/bin/mpifort
-- Check for working Fortran compiler: /home/cgw/miniforge3/envs/mfix-git/bin/mpifort - broken
CMake Error at /home/cgw/miniforge3/envs/mfix-git/share/cmake-3.28/Modules/CMakeTestFortranCompiler.cmake:59 (message):
  The Fortran compiler

    "/home/cgw/miniforge3/envs/mfix-git/bin/mpifort"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/tmp/TTT/build_dmp/CMakeFiles/CMakeScratch/TryCompile-38w6ll'

    Run Build Command(s): /home/cgw/miniforge3/envs/mfix-git/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_6144f/fast
    /usr/bin/gmake  -f CMakeFiles/cmTC_6144f.dir/build.make CMakeFiles/cmTC_6144f.dir/build
    gmake[1]: Entering directory '/tmp/TTT/build_dmp/CMakeFiles/CMakeScratch/TryCompile-38w6ll'
    Building Fortran object CMakeFiles/cmTC_6144f.dir/testFortranCompiler.f.o
    /home/cgw/miniforge3/envs/mfix-git/bin/mpifort    -c /tmp/TTT/build_dmp/CMakeFiles/CMakeScratch/TryCompile-38w6ll/testFortranCompiler.f -o CMakeFiles/cmTC_6144f.dir/testFortranCompiler.f.o
    Linking Fortran executable cmTC_6144f
    /home/cgw/miniforge3/envs/mfix-git/bin/cmake -E cmake_link_script CMakeFiles/cmTC_6144f.dir/link.txt --verbose=1
    /home/cgw/miniforge3/envs/mfix-git/bin/mpifort CMakeFiles/cmTC_6144f.dir/testFortranCompiler.f.o -o cmTC_6144f 
    /home/cgw/miniforge3/envs/mfix-git/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/cgw/miniforge3/envs/mfix-git/lib/libmpi_mpifh.so: undefined reference to `memcpy@GLIBC_2.14'
    /home/cgw/miniforge3/envs/mfix-git/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/cgw/miniforge3/envs/mfix-git/lib/./libpmix.so.2: undefined reference to `clock_gettime@GLIBC_2.17'
    collect2: error: ld returned 1 exit status
    gmake[1]: *** [CMakeFiles/cmTC_6144f.dir/build.make:99: cmTC_6144f] Error 1
    gmake[1]: Leaving directory '/tmp/TTT/build_dmp/CMakeFiles/CMakeScratch/TryCompile-38w6ll'
    gmake: *** [Makefile:127: cmTC_6144f/fast] Error 2

Installed packages

# packages in environment at /home/cgw/miniforge3/envs/mfix-git:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
aiohttp                   3.9.1           py310h2372a71_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
alabaster                 0.7.16             pyhd8ed1ab_0    conda-forge
alsa-lib                  1.2.10               hd590300_0    conda-forge
aom                       3.8.1                h59595ed_0    conda-forge
async-timeout             4.0.3              pyhd8ed1ab_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     23.2.0             pyh71513ae_0    conda-forge
babel                     2.14.0             pyhd8ed1ab_0    conda-forge
binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
blinker                   1.7.0              pyhd8ed1ab_0    conda-forge
blosc                     1.21.5               h0f2a231_0    conda-forge
brotli                    1.1.0                hd590300_1    conda-forge
brotli-bin                1.1.0                hd590300_1    conda-forge
brotli-python             1.1.0           py310hc6cd4ac_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.26.0               hd590300_0    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
cairo                     1.18.0               h3faef2a_0    conda-forge
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
cmake                     3.28.3               hcfe8598_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
contourpy                 1.2.0           py310hd41b1e2_0    conda-forge
curl                      8.5.0                hca28451_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
dav1d                     1.2.1                hd590300_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
docutils                  0.20.1          py310hff52083_3    conda-forge
double-conversion         3.3.0                h59595ed_0    conda-forge
eigen                     3.4.0                h00ab1b0_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
ffmpeg                    6.1.1           gpl_h8007c5b_104    conda-forge
ffmpeg-python             0.2.0                      py_0    conda-forge
flask                     3.0.2              pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_1    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.47.2          py310h2372a71_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
fribidi                   1.0.10               h36c2ea0_0    conda-forge
frozenlist                1.4.1           py310h2372a71_0    conda-forge
future                    0.18.3             pyhd8ed1ab_0    conda-forge
gcc                       13.2.0               h574f8da_2    conda-forge
gcc_impl_linux-64         13.2.0               h338b0a0_5    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
gfortran                  13.2.0               h0584b13_2    conda-forge
gfortran_impl_linux-64    13.2.0               h76e1118_5    conda-forge
git                       2.43.0          pl5321h7bc287a_0    conda-forge
gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.41             pyhd8ed1ab_0    conda-forge
gl2ps                     1.4.2                h0708190_0    conda-forge
glew                      2.1.0                h9c3ff4c_2    conda-forge
glib                      2.78.3               hfc55251_0    conda-forge
glib-tools                2.78.3               hfc55251_0    conda-forge
gmp                       6.3.0                h59595ed_0    conda-forge
gnutls                    3.7.9                hb077bed_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
gst-plugins-base          1.22.9               h8e1006c_0    conda-forge
gstreamer                 1.22.9               h98fc4e7_0    conda-forge
harfbuzz                  8.3.0                h3d44ed6_0    conda-forge
hdf4                      4.2.15               h2a13503_7    conda-forge
hdf5                      1.14.3          nompi_h4f84152_100    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
importlib-metadata        7.0.1              pyha770c72_0    conda-forge
itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
jsoncpp                   1.9.5                h4bd325d_1    conda-forge
kernel-headers_linux-64   2.6.32              he073ed8_16    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.5           py310hd41b1e2_1    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.16                 hb7c19ff_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20230802.1      cxx17_h59595ed_0    conda-forge
libaec                    1.1.2                h59595ed_1    conda-forge
libass                    0.17.1               h8fe9dca_1    conda-forge
libblas                   3.9.0           21_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcap                    2.69                 h0f662aa_0    conda-forge
libcblas                  3.9.0           21_linux64_openblas    conda-forge
libclang                  15.0.7          default_hb11cfb5_4    conda-forge
libclang13                15.0.7          default_ha2b6cf4_4    conda-forge
libcups                   2.3.3                h4637d8d_4    conda-forge
libcurl                   8.5.0                hca28451_0    conda-forge
libdeflate                1.19                 hd590300_0    conda-forge
libdrm                    2.4.114              h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-devel_linux-64     13.2.0             ha9c7c90_105    conda-forge
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgcrypt                 1.10.3               hd590300_0    conda-forge
libgfortran-ng            13.2.0               h69a702a_5    conda-forge
libgfortran5              13.2.0               ha4646dd_5    conda-forge
libglib                   2.78.3               h783c2da_0    conda-forge
libglu                    9.0.0             hac7e632_1003    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
libgpg-error              1.47                 h71f35ed_0    conda-forge
libhwloc                  2.9.3           default_h554bfaf_1009    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libidn2                   2.3.7                hd590300_0    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
liblapack                 3.9.0           21_linux64_openblas    conda-forge
libllvm15                 15.0.7               hb3ce162_4    conda-forge
libnetcdf                 4.9.2           nompi_h9612171_113    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnl                     3.9.0                hd590300_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.26          pthreads_h413a1c8_0    conda-forge
libopenvino               2023.3.0             h2e90f83_0    conda-forge
libopenvino-auto-batch-plugin 2023.3.0             hd5fc58b_0    conda-forge
libopenvino-auto-plugin   2023.3.0             hd5fc58b_0    conda-forge
libopenvino-hetero-plugin 2023.3.0             h3ecfda7_0    conda-forge
libopenvino-intel-cpu-plugin 2023.3.0             h2e90f83_0    conda-forge
libopenvino-intel-gpu-plugin 2023.3.0             h2e90f83_0    conda-forge
libopenvino-ir-frontend   2023.3.0             h3ecfda7_0    conda-forge
libopenvino-onnx-frontend 2023.3.0             hfbc7f12_0    conda-forge
libopenvino-paddle-frontend 2023.3.0             hfbc7f12_0    conda-forge
libopenvino-pytorch-frontend 2023.3.0             h59595ed_0    conda-forge
libopenvino-tensorflow-frontend 2023.3.0             h0bff32c_0    conda-forge
libopenvino-tensorflow-lite-frontend 2023.3.0             h59595ed_0    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpciaccess              0.17                 h166bdaf_0    conda-forge
libpng                    1.6.42               h2797004_0    conda-forge
libpq                     16.1                 h33b98f1_7    conda-forge
libprotobuf               4.25.1               hf27288f_1    conda-forge
libsanitizer              13.2.0               h7e041cc_5    conda-forge
libsndfile                1.2.2                hc60ed4a_1    conda-forge
libsqlite                 3.44.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libsystemd0               255                  h3516f8a_0    conda-forge
libtasn1                  4.19.0               h166bdaf_0    conda-forge
libtheora                 1.1.1             h7f98852_1005    conda-forge
libtiff                   4.6.0                ha9c0a0a_2    conda-forge
libunistring              0.9.10               h7f98852_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libuv                     1.46.0               hd590300_0    conda-forge
libva                     2.20.0               hd590300_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libvpx                    1.13.1               h59595ed_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libxkbcommon              1.6.0                hd429924_1    conda-forge
libxml2                   2.12.5               h232c23b_0    conda-forge
libzip                    1.10.1               h2629f0a_3    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
loguru                    0.7.2           py310hff52083_1    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
make                      4.3                  hd18ef5c_1    conda-forge
markupsafe                2.1.5           py310h2372a71_0    conda-forge
matplotlib                3.8.2           py310hff52083_0    conda-forge
matplotlib-base           3.8.2           py310h62c0568_0    conda-forge
mfix                      23.3.2                        0    https://private/conda/dist
mfix-doc                  23.3.2                        0    https://private/conda/dist
mfix-gui                  23.3.2                     py_0    https://private/conda/dist
mfix-solver               23.3.2               h3218e01_0    https://private/conda/dist
mfix-src                  23.3.2                        0    https://private/conda/dist
mpg123                    1.32.4               h59595ed_0    conda-forge
mpi                       1.0                     openmpi    conda-forge
mscorefonts               0.0.1                         3    conda-forge
multidict                 6.0.5           py310h2372a71_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.33               hf1915f5_6    conda-forge
mysql-libs                8.0.33               hca2cd23_6    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
nettle                    3.9.1                h7ab15ed_0    conda-forge
nlohmann_json             3.11.2               h27087fc_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.97                 h1d7d5a4_0    conda-forge
numpy                     1.26.4          py310hb13e2d6_0    conda-forge
ocl-icd                   2.3.1                h7f98852_0    conda-forge
ocl-icd-system            1.0.0                         1    conda-forge
openh264                  2.4.1                h59595ed_0    conda-forge
openjpeg                  2.5.0                h488ebb8_3    conda-forge
openmpi                   5.0.1              h4970cb7_101    conda-forge
openssl                   3.2.1                hd590300_0    conda-forge
p11-kit                   0.24.1               hc5aa10d_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pcre2                     10.42                hcad00b1_0    conda-forge
perl                      5.32.1          7_hd590300_perl5    conda-forge
pillow                    10.2.0          py310h01dd4db_0    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pixman                    0.43.2               h59595ed_0    conda-forge
ply                       3.11                       py_1    conda-forge
proj                      9.3.1                h1d62c97_0    conda-forge
psutil                    5.9.8           py310h2372a71_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pugixml                   1.14                 h59595ed_0    conda-forge
pulseaudio-client         16.1                 hb77b528_5    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.9          py310h04931ad_5    conda-forge
pyqt5-sip                 12.12.2         py310hc6cd4ac_5    conda-forge
pyqtgraph                 0.13.3             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.13         hd12c33a_1_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    4_cp310    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1           py310h2372a71_1    conda-forge
qt-main                   5.15.8              h450f30e_18    conda-forge
qtpy                      2.4.1              pyhd8ed1ab_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
rhash                     1.4.4                hd590300_0    conda-forge
setproctitle              1.3.3           py310h2372a71_0    conda-forge
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
simplejson                3.19.2          py310h2372a71_0    conda-forge
sip                       6.7.12          py310hc6cd4ac_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
sphinx                    7.2.6              pyhd8ed1ab_0    conda-forge
sphinx-prompt             1.4.0              pyhd8ed1ab_0    conda-forge
sphinx-substitution-extensions 2022.2.16                pypi_0    pypi
sphinx_rtd_theme          2.0.0              pyha770c72_0    conda-forge
sphinxcontrib-applehelp   1.0.8              pyhd8ed1ab_0    conda-forge
sphinxcontrib-devhelp     1.0.6              pyhd8ed1ab_0    conda-forge
sphinxcontrib-htmlhelp    2.0.5              pyhd8ed1ab_0    conda-forge
sphinxcontrib-jquery      4.1                pyhd8ed1ab_0    conda-forge
sphinxcontrib-jsmath      1.0.1              pyhd8ed1ab_0    conda-forge
sphinxcontrib-qthelp      1.0.7              pyhd8ed1ab_0    conda-forge
sphinxcontrib-serializinghtml 1.1.10             pyhd8ed1ab_0    conda-forge
sqlite                    3.44.2               h2c6b66d_0    conda-forge
svt-av1                   1.8.0                h59595ed_0    conda-forge
sysroot_linux-64          2.12                he073ed8_16    conda-forge
tbb                       2021.11.0            h00ab1b0_1    conda-forge
tbb-devel                 2021.11.0            h5ccd973_1    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tornado                   6.3.3           py310h2372a71_1    conda-forge
typing-extensions         4.9.0                hd8ed1ab_0    conda-forge
typing_extensions         4.9.0              pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
unicodedata2              15.1.0          py310h2372a71_0    conda-forge
urllib3                   2.2.0              pyhd8ed1ab_0    conda-forge
utfcpp                    4.0.5                ha770c72_0    conda-forge
vtk                       9.2.6           qt_py310h1234567_220    conda-forge
vtk-base                  9.2.6           qt_py310h1234567_220    conda-forge
vtk-io-ffmpeg             9.2.6           qt_py310h1234567_220    conda-forge
werkzeug                  3.0.1              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
wslink                    1.12.4             pyhd8ed1ab_0    conda-forge
x264                      1!164.3095           h166bdaf_2    conda-forge
x265                      3.5                  h924138e_3    conda-forge
xcb-util                  0.4.0                hd590300_1    conda-forge
xcb-util-image            0.4.0                h8ee46fc_1    conda-forge
xcb-util-keysyms          0.4.0                h8ee46fc_1    conda-forge
xcb-util-renderutil       0.3.9                hd590300_1    conda-forge
xcb-util-wm               0.4.1                h8ee46fc_1    conda-forge
xkeyboard-config          2.41                 hd590300_0    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.7                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-libxt                1.3.0                hd590300_1    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xf86vidmodeproto     2.3.1             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.9.4           py310h2372a71_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

Environment info

active environment : mfix-git
    active env location : /home/cgw/miniforge3/envs/mfix-git
            shell level : 1
       user config file : /home/cgw/.condarc
 populated config files : /home/cgw/miniforge3/.condarc
                          /home/cgw/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.38=0
                          __linux=6.6.8=0
                          __unix=0=0
       base environment : /home/cgw/miniforge3  (writable)
      conda av data dir : /home/cgw/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/cgw/miniforge3/pkgs
                          /home/cgw/.conda/pkgs
       envs directories : /home/cgw/miniforge3/envs
                          /home/cgw/.conda/envs
               platform : linux-64
             user-agent : conda/23.3.1 requests/2.31.0 CPython/3.10.12 Linux/6.6.8-gentoo gentoo/2.14 glibc/2.38
                UID:GID : 103:1000
             netrc file : None
           offline mode : False

[jakirkham]

Thanks for the report Charles! 🙏

This is likely because this feedstock is now building with a CentOS 7 image on linux-64, which uses GLIBC 2.17 (and has these newer symbols). So the packages don't work on systems using GLIBC older than 2.17

That said, the packages should carry this constraint to ensure that they are only installed on systems with a new enough GLIBC. However the GLIBC constraint wasn't being included before. PR ( https://github.com/conda-forge/openmpi-feedstock/pull/145 ) should fix that

An open question is whether older GLIBC are still of interest to maintain support for here. Will defer to the feedstock maintainers on that question

[jakirkham]

Actually sorry I misspoke, looking at the info provided above can see __glibc=2.38=0, which means this is on an even newer system than we used to build. So this should be supported and is a bug

It's possible adding sysroot (as done in the PR above) will fix this issue as well

[charlesgwaldman]

Thanks Jack. Yes, this was observed on an up-to-date Gentoo Linux system with current glibc 2.38. I have not previously had any issues with conda-forge packages on this system. Adding sysroot to the build deps sounds like the right solution (probably should be used by default for all packages, IMO)

[jakirkham]

The sysroot fix went in PR: https://github.com/conda-forge/openmpi-feedstock/pull/142

Packages are building and should be uploaded soon

Please look for a build/number of 3 on the resulting packages

[wfmu-listener]

I'm afraid that the problem persists in both OpenMPI 5.0.1 and 5.0.2 as packaged by conda-forge. The last usable version is 5.0.0

The same failure is observed with both a very recent system (Gentoo linux with glibc 2.39) and and older system (CentOS7 with glibc 2.17)

Both complain about 'memcpy@GLIBC_2.14' and 'clock_gettime@GLIBC_2.17'. (These versioned symbols sure are a pain).

centos7-log.txt gentoo-log.txt

[charlesgwaldman]

That last comment was from me, I was logged in under a different GitHub account without realizing.

One question I have is - what changed between OpenMPI 5.0.0 and 5.0.1? Can we just go back to the way 5.0.0 was getting built? That version does not suffer from the portability problems.

[dalcinl]

The most likely problem is that the the 5.0.1 image was built in a different, newer docker image. You should probably as the core conda-forge team. This is ultimately not an Open MPI issue or the fault of this feedstock (although I could be partially wrong).

[charlesgwaldman]

Thanks dalcinl. How do I bring this to the attention of the core conda-forge team?

[dalcinl]

I usually contact the team via Gitter https://conda-forge.org/community/getting-in-touch/#gitter-and-element Ask them and let's see what they say. Maybe we can use some hack to make the binaries use the older symbol versions.

[dalcinl]

@jakirkham Do you think we are somehow messing things up in this feedstock?

[jakirkham]

The issue they are seeing is they are on newer systems (GLIBC 2.28+) and are having trouble resolving symbols that should be available on their systems as we built on (GLIBC 2.17+)

IOW the symbols should be available in their cases, but for some reason they are not

The bug may very well be in our build, but am a little fuzzy on how it is occurring

[jakirkham]

Could one of you seeing this error please trying installing sysroot_linux-64=2.17 (assuming you are on Linux x86_64) and let us know if you still see issues?

[jakirkham]

Tried adding the GLIBC constraint to those packages directly ( https://github.com/conda-forge/openmpi-feedstock/pull/147 ). Maybe that helps?

[charlesgwaldman]

@jakirkham Installing sysroot_linux-64=2.17 resolved the issue on both CentOS7 and Gentoo. Thank you!

[charlesgwaldman]

It also works with sysroot_linux=2.28. The problem was caused by sysroot_linux-2.12 which got installed when I installed gfortran. Installing openmpi doesn't install a sysroot at all. So if you do mamba install gfortran openmpi in a new environment, you will wind up with the 2.12 sysroot and an openmpi which requires 2.17 or newer.

[jakirkham]

Thanks Charles! 🙏

Yeah that's what I was wondering about

Then I think we should try PR: https://github.com/conda-forge/openmpi-feedstock/pull/147

[jakirkham]

New packages are building. Will probably be a bit before they upload and mirror to CDN

Please test out tomorrow and let us know how it goes

[charlesgwaldman]

It's still broken, or else I'm not seeing new packages. Testing this should be very easy. Just do mamba install openmpi gfortran in a new environment and see which sysroot gets pulled in. If it's sysroot_linux-64 2.12 then we still have a problem because that's incompatible with mpifort.

bash$ mamba create -n test; mamba activate test
(test)$ mamba install openmpi gfortran
  Package                     Version  Build                  Channel           Size
──────────────────────────────────────────────────────────────────────────────────────
  Install:
──────────────────────────────────────────────────────────────────────────────────────

  + mpi                           1.0  openmpi                conda-forge     Cached
  + _libgcc_mutex                 0.1  conda_forge            conda-forge     Cached
  + libstdcxx-ng               13.2.0  h7e041cc_5             conda-forge     Cached
  + ld_impl_linux-64             2.40  h41732ed_0             conda-forge     Cached
  + ca-certificates          2024.2.2  hbcca054_0             conda-forge     Cached
  + libgomp                    13.2.0  h807b86a_5             conda-forge     Cached
  + _openmp_mutex                 4.5  2_gnu                  conda-forge     Cached
  + libgcc-ng                  13.2.0  h807b86a_5             conda-forge     Cached
  + libiconv                     1.17  hd590300_2             conda-forge     Cached
  + libsanitizer               13.2.0  h7e041cc_5             conda-forge     Cached
  + openssl                     3.2.1  hd590300_1             conda-forge     Cached
  + icu                          73.2  h59595ed_0             conda-forge     Cached
  + xz                          5.2.6  h166bdaf_0             conda-forge     Cached
  + libzlib                    1.2.13  hd590300_5             conda-forge     Cached
  + libgfortran5               13.2.0  ha4646dd_5             conda-forge     Cached
  + libnl                       3.9.0  hd590300_0             conda-forge     Cached
  + libevent                   2.1.12  hf998b51_1             conda-forge     Cached
  + libxml2                    2.12.6  h232c23b_1             conda-forge     Cached
  + libgfortran-ng             13.2.0  h69a702a_5             conda-forge     Cached
  + libhwloc                    2.9.3  default_h554bfaf_1009  conda-forge     Cached
  + openmpi                     5.0.3  h7fc1de5_100           conda-forge       15MB
  + libgcc-devel_linux-64      13.2.0  ha9c7c90_105           conda-forge     Cached
  + kernel-headers_linux-64    2.6.32  he073ed8_17            conda-forge     Cached
  + sysroot_linux-64             2.12  he073ed8_17            conda-forge     Cached
  + binutils_impl_linux-64       2.40  hf600244_0             conda-forge     Cached
  + gcc_impl_linux-64          13.2.0  h338b0a0_5             conda-forge     Cached
  + gfortran_impl_linux-64     13.2.0  h76e1118_5             conda-forge     Cached
  + gcc                        13.2.0  hd6cf55c_3             conda-forge     Cached
  + gfortran                   13.2.0  h98b45c4_3             conda-forge     Cached
....
(test)$ mpifort /tmp/test_mpi.f90 
/home/cgw/miniforge3/envs/test/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/cgw/miniforge3/envs/test/lib/libmpi_mpifh.so: undefined reference to `memcpy@GLIBC_2.14'
/home/cgw/miniforge3/envs/test/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/cgw/miniforge3/envs/test/lib/./libpmix.so.2: undefined reference to `clock_gettime@GLIBC_2.17'
collect2: error: ld returned 1 exit status

[charlesgwaldman]

(test)$ cat /tmp/test_mpi.f90
      program hello
      use mpi_f08
      implicit none
      integer(kind=MPI_INTEGER_KIND) ierror
      call MPI_INIT(ierror)
      call MPI_FINALIZE(ierror)
      end program

[leofang]

Could you show the full output? I don't see the mpifort package in the list, for example

[charlesgwaldman]

That is the full package list, the package is called openmpi

[charlesgwaldman]

(test)$ mamba info

          mamba version : 1.5.8
     active environment : test
    active env location : /home/cgw/miniforge3/envs/test
            shell level : 1
       user config file : /home/cgw/.condarc
 populated config files : /home/cgw/miniforge3/.condarc
                          /home/cgw/.condarc
          conda version : 24.3.0
    conda-build version : not installed
         python version : 3.10.14.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=24.3.0=0
                          __glibc=2.39=0
                          __linux=6.8.2=0
                          __unix=0=0
       base environment : /home/cgw/miniforge3  (writable)
      conda av data dir : /home/cgw/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/cgw/miniforge3/pkgs
                          /home/cgw/.conda/pkgs
       envs directories : /home/cgw/miniforge3/envs
                          /home/cgw/.conda/envs
               platform : linux-64
             user-agent : conda/24.3.0 requests/2.31.0 CPython/3.10.14 Linux/6.8.2-gentoo gentoo/2.15 glibc/2.39 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
                UID:GID : 103:1000
             netrc file : None
           offline mode : False

(test)$ mamba list
# packages in environment at /home/cgw/miniforge3/envs/test:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
gcc                       13.2.0               hd6cf55c_3    conda-forge
gcc_impl_linux-64         13.2.0               h338b0a0_5    conda-forge
gfortran                  13.2.0               h98b45c4_3    conda-forge
gfortran_impl_linux-64    13.2.0               h76e1118_5    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
kernel-headers_linux-64   2.6.32              he073ed8_17    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libgcc-devel_linux-64     13.2.0             ha9c7c90_105    conda-forge
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgfortran-ng            13.2.0               h69a702a_5    conda-forge
libgfortran5              13.2.0               ha4646dd_5    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
libhwloc                  2.9.3           default_h554bfaf_1009    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
libnl                     3.9.0                hd590300_0    conda-forge
libsanitizer              13.2.0               h7e041cc_5    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libxml2                   2.12.6               h232c23b_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
mpi                       1.0                     openmpi    conda-forge
openmpi                   5.0.3              h7fc1de5_100    conda-forge
openssl                   3.2.1                hd590300_1    conda-forge
sysroot_linux-64          2.12                he073ed8_17    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge

[leofang]

Sorry I meant openmpi-mpifort not mpiport. You need to have it installed too, otherwise it seems you're using the system-provided MPI compiler wrapper, not the one coming from conda-forge (which the fix was applied to).

[charlesgwaldman]

What?

bash$ which mpifort
which: no mpifort in (/home/cgw/Applications/.bin:/home/cgw/miniforge3/condabin:/home/cgw/bin:/home/cgw/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/18/bin:/usr/lib/llvm/17/bin)

bash$ mamba activate test
(test)$  which mpifort
/home/cgw/miniforge3/envs/test/bin/mpifort

I have been using openmpi from conda-forge for several years and have never installed or heard of openmpi-mpifort until now.

[charlesgwaldman]

@leofang As far as I can tell the difference between installing openmpi and gfortran, vs installing openmpi-mpifort is that openmpi-mpifort is tied to gfortran 11, while openmpi+gfortran gives you gfotran 13.

What is the purpose of the openmpi-mpifort package? Is there some documentation that says why I have to have this installed?

[leofang]

What is the purpose of the openmpi-mpifort package? Is there some documentation that says why I have to have this installed?

The compiler wrapper packages openmpi-{mpicc,mpicxx,mpifort} have been there forever (way before I became a maintainer IIRC). The purpose is to ensure a consistent compiler toolchain (same version as openmpi used at build time) is installed, so as to avoid potential ABI issues.

It appears that when using CUDA 11 to build (which is what this feedstock does) we're pinned at gfortran 11: https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/f2335dfd386a8ef51f75676bf24b74efe7aeab93/recipe/conda_build_config.yaml#L44 Even if we migrate to CUDA 12, we'd be using gfortran 12, not 13: https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/f2335dfd386a8ef51f75676bf24b74efe7aeab93/recipe/migrations/cuda120.yaml#L91 Unless there is a global migrator that moves us to gfortran 13, there's not much we can do in this feedstock. I suggest opening an issue in https://github.com/conda-forge/conda-forge.github.io to discuss.

What would be the reason that you want to ignore the ABI compatibility issue and use gfortran 13?

[charlesgwaldman]

Thanks for the reply.

It's not so much that I want to ignore ABI compatibility, but that I've been maintaining a Conda package that uses OpenMPI and gfortran for several years, and we have used the mpifort that comes with the openmpi package and have had no issues. It's not clear to users (e.g. myself) why openmpi-mpifort is needed, when an mpifort binary is provided by the openmpi package itself. If this mpifort is known to be broken or incompatible in some way, why is it being distributed with openmpi?

[alphaparrot]

Just an update that this is also an issue with clean installs of Linux Mint 21.3 (based on Ubuntu 24.04), using the most recent versions of openmpi and gcc/gfortran (version 13), as noted above for other versions. Generally it would be nice to be able to use this package with up-to-date compilers, so that packages and libraries we distribute which use openmpi can be used on systems with the latest compilers (especially seeing as how openmpi applications often find themselves on HPC clusters, where users often don't necessarily have a ton of control over compiler versions and sysadmins may be reluctant to have many installed versions of compilers). I would like to second the above question; is the current advice that this package should not be used, and instead only ever openmpi-mpifort? At least until this versioned symbols issue is fixed?

[LourensVeen]

(This reply is a lot shorter if you ignore the code blocks and read the text in between, I'm just trying to create a clear reproducer. Or scroll to the bottom for the conclusion.)

Reproducing

I'm running into this same issue (albeit in C, but it's identical): installing openmpi and gcc simultaneously results in the above mentioned linking errors because gcc pulls in sysroot_linux-64 2.12, which then shadows the system glibc in the dynamic linker lookup, causing the error because openmpi requires something newer.

$ conda create -n openmpi-test
$ conda activate openmpi-test
$ conda install openmpi
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/lourens/.miniconda3/envs/openmpi-test

  added / updated specs:
    - openmpi

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge 
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu 
  ca-certificates    conda-forge/linux-64::ca-certificates-2024.2.2-hbcca054_0 
  icu                conda-forge/linux-64::icu-73.2-h59595ed_0 
  libevent           conda-forge/linux-64::libevent-2.1.12-hf998b51_1 
  libgcc-ng          conda-forge/linux-64::libgcc-ng-13.2.0-h77fa898_7 
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-13.2.0-h69a702a_7 
  libgfortran5       conda-forge/linux-64::libgfortran5-13.2.0-hca663fb_7 
  libgomp            conda-forge/linux-64::libgomp-13.2.0-h77fa898_7 
  libhwloc           conda-forge/linux-64::libhwloc-2.10.0-default_h5622ce7_1001 
  libiconv           conda-forge/linux-64::libiconv-1.17-hd590300_2 
  libnl              conda-forge/linux-64::libnl-3.9.0-hd590300_0 
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-13.2.0-hc0a3c3a_7 
  libxml2            conda-forge/linux-64::libxml2-2.12.7-hc051c1a_0 
  libzlib            conda-forge/linux-64::libzlib-1.3.1-h4ab18f5_1 
  mpi                conda-forge/linux-64::mpi-1.0-openmpi 
  openmpi            conda-forge/linux-64::openmpi-5.0.3-h47314c5_102 
  openssl            conda-forge/linux-64::openssl-3.3.0-h4ab18f5_3 
  xz                 conda-forge/linux-64::xz-5.2.6-h166bdaf_0 

# <snip>

This installs an mpicc, even though we don't have openmpi-mpicc:

$ which mpicc
/home/lourens/.miniconda3/envs/openmpi-test/bin/mpicc

and it installs a libprrte.so that uses symbols that require glibc>2.12:

$ nm -CD ~/.miniconda3/envs/openmpi-test/lib/libprrte.so.3.0.5 | grep memcpy
                 U memcpy@GLIBC_2.14
                 U __memcpy_chk@GLIBC_2.3.4

Since we don't have a Conda-installed glibc, the system one is used:

$ ldd ~/.miniconda3/envs/openmpi-test/lib/libprrte.so.3.0.5
        # <snip>
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc11b6da000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc11bd26000)
        # <snip>

and since that is new enough, the dependencies can be resolved and everything works (note that this uses mpicc from Conda openmpi, with system gcc because there's no Conda GCC installed):

$ cat test_conda_openmpi.c 
#include <mpi.h>

int main(int argc, char **argv) {
    MPI_Init(&argc, &argv);
    MPI_Finalize();
}

$ mpicc -o test_conda_openmpi test_conda_openmpi.c
# <no error>

Installing gcc pulls in sysroot_linux-64 2.12:

$ conda install gcc
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/lourens/.miniconda3/envs/openmpi-test

  added / updated specs:
    - gcc

The following NEW packages will be INSTALLED:

  binutils_impl_lin~ conda-forge/linux-64::binutils_impl_linux-64-2.40-ha1999f0_1 
  gcc                conda-forge/linux-64::gcc-13.2.0-hc7bed06_7 
  gcc_impl_linux-64  conda-forge/linux-64::gcc_impl_linux-64-13.2.0-h9eb54c0_7 
  kernel-headers_li~ conda-forge/noarch::kernel-headers_linux-64-2.6.32-he073ed8_17 
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.40-hf3520f5_1 
  libgcc-devel_linu~ conda-forge/noarch::libgcc-devel_linux-64-13.2.0-hceb6213_107 
  libsanitizer       conda-forge/linux-64::libsanitizer-13.2.0-h6ddb7a1_7 
  sysroot_linux-64   conda-forge/noarch::sysroot_linux-64-2.12-he073ed8_17 

and this breaks things:

$ mpicc -o test_conda_openmpi test_conda_openmpi.c
/home/lourens/.miniconda3/envs/openmpi-test/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/lourens/.miniconda3/envs/openmpi-test/lib/libmpi.so: undefined reference to `memcpy@GLIBC_2.14'
/home/lourens/.miniconda3/envs/openmpi-test/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/lourens/.miniconda3/envs/openmpi-test/lib/./libpmix.so.2: undefined reference to `clock_gettime@GLIBC_2.17'
collect2: error: ld returned 1 exit status

I can confirm that installing openmpi-mpicc fixes it, but seeming only because it causes a downgrade to a previous version of openmpi which was probably built in an older container image that doesn't have the newer glibc:

$ conda install openmpi-mpicc
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/lourens/.miniconda3/envs/openmpi-test

  added / updated specs:
    - openmpi-mpicc

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    gcc-12.3.0                 |       h915e2ae_7          25 KB  conda-forge
    gcc_impl_linux-64-12.3.0   |       h58ffeeb_7        48.8 MB  conda-forge
    libgcc-devel_linux-64-12.3.0|     h0223996_107         2.4 MB  conda-forge
    libsanitizer-12.3.0        |       hb8811af_7         3.8 MB  conda-forge
    openmpi-5.0.1              |     hb0ee255_100        14.3 MB  conda-forge
    openmpi-mpicc-5.0.1        |     hd590300_100          12 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        69.3 MB

The following NEW packages will be INSTALLED:

  binutils_linux-64  conda-forge/linux-64::binutils_linux-64-2.40-hdade7a5_3 
  gcc_linux-64       conda-forge/linux-64::gcc_linux-64-12.3.0-h6477408_3 
  openmpi-mpicc      conda-forge/linux-64::openmpi-mpicc-5.0.1-hd590300_100 

The following packages will be DOWNGRADED:

  gcc                                     13.2.0-hc7bed06_7 --> 12.3.0-h915e2ae_7 
  gcc_impl_linux-64                       13.2.0-h9eb54c0_7 --> 12.3.0-h58ffeeb_7 
  libgcc-devel_linu~                    13.2.0-hceb6213_107 --> 12.3.0-h0223996_107 
  libhwloc                     2.10.0-default_h5622ce7_1001 --> 2.9.3-default_h554bfaf_1009 
  libsanitizer                            13.2.0-h6ddb7a1_7 --> 12.3.0-hb8811af_7 
  libzlib                                  1.3.1-h4ab18f5_1 --> 1.2.13-h4ab18f5_6 
  openmpi                                5.0.3-h47314c5_102 --> 5.0.1-hb0ee255_100 

and now we have older symbols that work with glibc==2.12:

$ nm ~/.miniconda3/pkgs/openmpi-5.0.1-hb0ee255_100/lib/libprrte.so.3.0.3 | grep memcpy
                 U __memcpy_chk@GLIBC_2.3.4
                 U memcpy@GLIBC_2.2.5

If I specifically ask for the latest openmpi-mpicc then it's broken again:

$ conda install 'openmpi-mpicc==5.0.3'
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/lourens/.miniconda3/envs/openmpi-test

  added / updated specs:
    - openmpi-mpicc==5.0.3

The following packages will be UPDATED:

  libhwloc                      2.9.3-default_h554bfaf_1009 --> 2.10.0-default_h5622ce7_1001 
  openmpi                                5.0.1-hb0ee255_100 --> 5.0.3-h47314c5_102 
  openmpi-mpicc                          5.0.1-hd590300_100 --> 5.0.3-hc43e4ee_102 

The following packages will be DOWNGRADED:

  gcc                                     12.3.0-h915e2ae_7 --> 11.4.0-h602e360_7 
  gcc_impl_linux-64                       12.3.0-h58ffeeb_7 --> 11.4.0-h00c12a0_7 
  gcc_linux-64                            12.3.0-h6477408_3 --> 11.4.0-h0f0c6b6_3 
  libgcc-devel_linu~                    12.3.0-h0223996_107 --> 11.4.0-h515aa5d_107 
  libsanitizer                            12.3.0-hb8811af_7 --> 11.4.0-h5763a12_7 

Indeed if we try to compile:

$ mpicc -o test_conda_openmpi test_conda_openmpi.c
/home/lourens/.miniconda3/envs/openmpi-test/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/lourens/.miniconda3/envs/openmpi-test/lib/libmpi.so: undefined reference to `memcpy@GLIBC_2.14'
/home/lourens/.miniconda3/envs/openmpi-test/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/lourens/.miniconda3/envs/openmpi-test/lib/./libpmix.so.2: undefined reference to `clock_gettime@GLIBC_2.17'
collect2: error: ld returned 1 exit status

Why it's still broken (?)

Despite #142 and #145, it seems that no released package ever had a dependency on sysroot:

$ conda search --info openmpi-mpicc | grep sysroot
$ conda search --info openmpi | grep sysroot
$ conda search --info openmpi-mpifort | grep sysroot

Potential solution?

It seems to me that depending on sysroot isn't the right solution anyway, __glibc will do fine if it's new enough. Instead, it seems to me that openmpi, and in fact all packages built with the newer docker image that contains a newer glibc, should have an automatic run_constrained requirement on sysroot that ensures that if sysroot is installed, it's new enough.

That suggests that this issue should be resolved elsewhere, but I don't know enough about how conda-forge works to know where to take it. Can someone give a hint?

[dalcinl]

Maybe this is actually some sort of build issue? If the package had been built with the proper sysroot installed in the build environment, then the binaries would not end up using newer symbols. Are we somehow missing something in our recipe to constrain the build-time sysroot? Or is really adding a run_constrained for sysroot the proper way to go?

[LourensVeen]

Interesting question. Does conda-forge have a policy on which minimum glibc it supports? Then packages should adhere to that, and then the Docker image should provide that exact glibc to link against, either natively or via a sysroot package. Relying on each and every package maintainer to read that policy and adjust their package definition accordingly doesn't sound like a working strategy...

Glibc 2.12 was released in 2010, and 2.17 in 2012, so even enterprise Linux distributions should have at least 2.17 by now and requiring it doesn't seem all that controversial to me. So perhaps another question is why conda install gcc drags in a 14 year old glibc even though a 12 year old one is available?

[LourensVeen]

Ah, maybe the answer to that second question is that this causes packages to be built against glibc 2.12, thus making them compatible with everything and avoiding the problem we're seeing?

But the openmpi package has compiler('c') as the compiler, which is gcc on Linux, which should then pull in the sysroot 2.12. And now I understand your comment :smile:. Yes, this is strange then.

[LourensVeen]

And looking at the build configuration, this package explicitly builds against glibc 2.17, first by explicitly having sysroot 2.17 as a build dependency, and after this commit in conda_build_config.yaml.

Searching for c_stdlib_version led me to this announcement which finally sheds some light on things. It links to https://github.com/conda-forge/conda-forge.github.io/issues/2102. I've been so bold as to add a comment there.

[xylar]

@conda-forge/openmpi, we're seeing this issue in https://github.com/conda-forge/esmf-feedstock/pull/116. I tried adding openmpi-mpifort to the build section of the recipe and it only made things worse For cross-compiling, we're seeing errors like:

gcc_linux-64 11.4.0 h0f0c6b6_2 needed by gfortran_linux-64-11.4.0-h8f970dc_2

suggesting that it's maybe getting confused about what architecture to install for. And it made no different for the linux-64 build, which still has:

mpif90   -m64 -mcmodel=small -pthread -Wl,--no-as-needed -fopenmp -L$SRC_DIR/lib/libO/Linux.gfortran.64.openmpi.default -L$PREFIX/lib -L$PREFIX/lib -L$BUILD_PREFIX/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/ -Wl,-rpath,$SRC_DIR/lib/libO/Linux.gfortran.64.openmpi.default -Wl,-rpath,$PREFIX/lib  -Wl,-rpath,$PREFIX/lib -Wl,-rpath,$BUILD_PREFIX/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/ -o $SRC_DIR/test/testO/Linux.gfortran.64.openmpi.default/ESMF_StringUTest ESMCI_StringSubr.o  ESMF_StringUTest.o -lesmf  -lrt -lstdc++ -ldl -lnetcdff -lnetcdf -lpioc
/home/conda/feedstock_root/build_artifacts/esmf_1717757906123/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/esmf_1717757906123/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/./libpmix.so.2: undefined reference to `clock_gettime@GLIBC_2.17'
/home/conda/feedstock_root/build_artifacts/esmf_1717757906123/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/conda/feedstock_root/build_artifacts/esmf_1717757906123/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh/lib/libmpi_mpifh.so: undefined reference to `memcpy@GLIBC_2.14'
collect2: error: ld returned 1 exit status

Here is the constents of the build environment for linux-64, which I think is the most relevant:

  environment location: /home/conda/feedstock_root/build_artifacts/esmf_1717757906123/_build_env

The following NEW packages will be INSTALLED:

    _libgcc_mutex:            0.1-conda_forge             conda-forge
    _openmp_mutex:            4.5-2_gnu                   conda-forge
    binutils_impl_linux-64:   2.40-ha1999f0_2             conda-forge
    binutils_linux-64:        2.40-hdade7a5_3             conda-forge
    bzip2:                    1.0.8-hd590300_5            conda-forge
    c-ares:                   1.28.1-hd590300_0           conda-forge
    ca-certificates:          2024.6.2-hbcca054_0         conda-forge
    cmake:                    3.29.5-hcafd917_0           conda-forge
    gcc_impl_linux-64:        12.3.0-h58ffeeb_7           conda-forge
    gcc_linux-64:             12.3.0-h6477408_3           conda-forge
    gfortran_impl_linux-64:   12.3.0-h1645026_7           conda-forge
    gfortran_linux-64:        12.3.0-h617cb40_3           conda-forge
    gnuconfig:                2020.11.07-hd8ed1ab_0       conda-forge
    gxx_impl_linux-64:        12.3.0-h2a574ab_7           conda-forge
    gxx_linux-64:             12.3.0-h4a1b8e8_3           conda-forge
    icu:                      73.2-h59595ed_0             conda-forge
    kernel-headers_linux-64:  2.6.32-he073ed8_17          conda-forge
    keyutils:                 1.6.1-h166bdaf_0            conda-forge
    krb5:                     1.21.2-h659d440_0           conda-forge
    ld_impl_linux-64:         2.40-hf3520f5_2             conda-forge
    libcurl:                  8.8.0-hca28451_0            conda-forge
    libedit:                  3.1.20191231-he28a2e2_2     conda-forge
    libev:                    4.33-hd590300_2             conda-forge
    libevent:                 2.1.12-hf998b51_1           conda-forge
    libexpat:                 2.6.2-h59595ed_0            conda-forge
    libgcc-devel_linux-64:    12.3.0-h0223996_107         conda-forge
    libgcc-ng:                13.2.0-h77fa898_7           conda-forge
    libgfortran-ng:           13.2.0-h69a702a_7           conda-forge
    libgfortran5:             13.2.0-hca663fb_7           conda-forge
    libgomp:                  13.2.0-h77fa898_7           conda-forge
    libhwloc:                 2.9.3-default_h554bfaf_1009 conda-forge
    libiconv:                 1.17-hd590300_2             conda-forge
    libnghttp2:               1.58.0-h47da74e_1           conda-forge
    libnl:                    3.9.0-hd590300_0            conda-forge
    libsanitizer:             12.3.0-hb8811af_7           conda-forge
    libssh2:                  1.11.0-h0841786_0           conda-forge
    libstdcxx-devel_linux-64: 12.3.0-h0223996_107         conda-forge
    libstdcxx-ng:             13.2.0-hc0a3c3a_7           conda-forge
    libuv:                    1.48.0-hd590300_0           conda-forge
    libxcrypt:                4.4.36-hd590300_1           conda-forge
    libxml2:                  2.12.7-hc051c1a_1           conda-forge
    libzlib:                  1.3.1-h4ab18f5_1            conda-forge
    make:                     4.3-hd18ef5c_1              conda-forge
    mpi:                      1.0-openmpi                 conda-forge
    ncurses:                  6.5-h59595ed_0              conda-forge
    openmpi:                  5.0.1-hb0ee255_100          conda-forge
    openmpi-mpifort:          5.0.1-heb67821_100          conda-forge
    openssl:                  3.3.1-h4ab18f5_0            conda-forge
    perl:                     5.32.1-7_hd590300_perl5     conda-forge
    pkg-config:               0.29.2-h36c2ea0_1008        conda-forge
    rhash:                    1.4.4-hd590300_0            conda-forge
    sysroot_linux-64:         2.12-he073ed8_17            conda-forge
    xz:                       5.2.6-h166bdaf_0            conda-forge
    zstd:                     1.5.6-ha6fb4c9_0            conda-forge

Notably sysroot 2.12.

I'm at a bit of a loss and can't say I was able to follow the discussion above particularly well. Is there a suggested fix for this?

[LourensVeen]

I have upgraded my meta.yaml to the new way of specifying use of the C library using

requirements:
  build:
    - {{ stdlib('c') }}
    - <other stuff>

and then in my conda_build_config.yaml I have:

c_stdlib:
  - sysroot                    # [linux]
  - macos_deployment_target    # [osx]

c_stdlib_version:
  - 2.17                       # [linux]
  - 11.3                       # [osx]

I have yet to get a Mac to try the osx build on, so I'm not sure that that works yet, but it fixes the Linux build at least.

What this does is to specify that we want at least glibc 2.17 for the package we're building, which ensures that glibc 2.17 is available so that we can link to it properly. Of course this is strictly speaking incorrect if the present package works with 2.12, but since 2.17 will soon be the standard for everything anyway, it doesn't matter much.

[xylar]

@LourensVeen, thank you! I'll give that a shot.

[minrk]

I'm not sure why building with sysroot 2.17 creates a package that claims to be compatible with sysroot 2.12, or if it's something particular about openmpi that isn't universal. This comes up in this package's own tests, where sysroot 2.12 is installed in the test environment. I don't know why that is.

FWIW, compiling with $LDFLAGS avoids this problem, at least in openmpi's own tests (#159)

mpif77 test.f # fails to find versioned glib symbols
mpif77 $LDFLAGS test.f  # succeeds

I'm not actually sure which LDFLAG is responsible (maybe -Wl,--as-needed?) There does seem to be a problem in general where this package being built with a particular sysroot has a runtime conflict with other packages being built with lower sysroot, but the package dependencies don't actually capture this conflict. I don't know enough about all this linking to say what's the right fix and whether it belongs in sysroot itself or openmpi.

Does it make sense to put sysroot 2.17 in run_constrained for this package? And if so, do we need repodata patches? And does it make sense to do that in general in sysroot itself, or just here?

[minrk]

From this comment, it appears the missing flag is:

-Wl,--allow-shlib-undefined

maybe we should try to get this into the compiler wrapper's default flags

[minrk]

159 appears to confirm that what was missing was -Wl,--allow-shlib-undefined.

I think setting:

export OMPI_LDFLAGS="-L$PREFIX/lib -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib"

will fix anything using the compiler wrapper and building with sysroot 2.12, and #159 will make sure that's the default, I think actually resolving this issue.

In general, anything using $LDFLAGS (everything should) should not encounter this problem, but notably I believe that CMake's FindMPI does not do this. I'm not sure what arguments need to be specified, possibly `-DMPI_FORTRAN_LINK_FLAGS="-L$PREFIX/lib -Wl,--allow-shlib-undefined -Wl,-rpath,$PREFIX/lib" but I'm not sure.

[LourensVeen]

I agree that that would probably (I didn't check it) avoid the build failure, but it wouldn't actually solve the problem, just disable the error at that particular point and kick the can down the road.

The built package would still have a dependency on openmpi, which would still depend on glibc 2.17, and things would still break if the user installs gcc as well as the built package, because gcc would drag in sysroot==2.12, which would then be used by the dynamic linker loader when trying to start an MPI program, and it would fail to find the newer symbols there.

[minrk]

I don't want to speak with too much confidence because I'm a bit out of my depth, but I don't think that's the case. For example, in https://github.com/conda-forge/openmpi-feedstock/pull/159, the test environments for opempi-mpicc (and fort, etc.) pull in gcc and sysroot==2.12 and I think exercise precisely the case you describe. They compile and link correctly (thanks to the addition of -Wl,--allow-shlib-undefined), and then execute correctly because the glibc used at runtime is not the backdated one that's linked from sysroot. Runtime glibc comes from the system (which would have prevented installation if it really was too new, due to the __glibc requirement). I'm not sure my description of the situation is completely right, but I do know that an env with sysroot=2.12 does not prevent execution when linking symbols from 2.17.

[leofang]

Seems like a big hammer to me. I feel we need alignment with @conda-forge/core regarding the discussion https://github.com/conda-forge/linux-sysroot-feedstock/issues/63 started by @h-vetinari (EDIT: I see @minrk had also raised the discussion here https://github.com/conda-forge/conda-forge.github.io/issues/2102#issuecomment-2156641517). I am not comfortable adding this flag unconditionally. It only solves our problem but I do not believe Open MPI is the only project affected by glibc symbol issues.

[isuruf]

I do not believe Open MPI is the only project affected by glibc symbol issues.

It's the only package that compiles other things and does not respect LDFLAGS.

[LourensVeen]

@minrk: Good point (and a working example is hard to argue with :smile:). I failed to consider that the dynamic linker would come from Conda and find the sysroot-installed glibc, but that the loader comes from the system and wouldn't be bothered by it.

Just disabling the check still feels wrong to me though, and I'm worried that it could cause other problems. What if you tried to compile some MPI code that itself needs a newer glibc than you have available on your system? I guess it would fail when you try to run the program, but it might be hard to debug, and you would expect it to fail at link time.

This could affect Conda too if an MPI-using package got an update that introduced a dependency on a newer glibc than Conda uses (say 2.25). You probably wouldn't notice outside of Conda (because you have a much newer glibc everywhere), and the package build would succeed (because the check is disabled), but a user trying to run the package on an older system with glibc>=2.17<2.25 would still get an error. I guess we'd have to hope there's a test that actually runs things and fails the build that way.

[minrk]

It's the only package that compiles other things and does not respect LDFLAGS.

FWIW, it's cmake's FindMPI, not openmpi, that is ignoring compiler flags. I think the mistake I made in #158 is to assume that the mpi compiler wrappers should be responsible for respecting $CFLAGS/$LDFLAGS, but that's not right - they should be minimal extensions of $CC/$FC etc. with just enough to find/link mpi.h/libmpi. $CFLAGS/$LDFLAGS should be passed to the compiler wrappers, just like regular compilers, which happens as expected after FindMPI succeeds, but not during for some reason. Sounds plausibly like a CMake bug to me.

Within the context of conda-forge where the linked glibc may be older than that of your dependencies, -Wl,--allow-shlib-undefined is part of what's required to link conda-forge openmpi (or ~any other dependency using 2.17).

a user trying to run the package on an older system with glibc>=2.17<2.25 would still get an error

@LourensVeen I think the dependency on __glibc version already addresses that. The package wouldn't be installable for them because it would have __glibc >=2.25 in its requirements (from sysroot's run_exports here). Instead, they would get the latest build that supported their version of glibc.

Also, note that disabling the check in the openmpi compiler wrapper is just applying a subset of what's already applied to all conda-forge-built shared libraries via $LDFLAGS. It's just that for some reason FindMPI checks that mpicc and friends work without $LDFLAGS, and this is the only flag where that really matters.

[LourensVeen]

I was thinking of a scenario where the packager wouldn't notice the new dependency, and fail to update the sysroot dependency. But that's arguably a bug in that package then, and anyway it probably wouldn't build on conda-forge to begin with, because the Docker container doesn't have the new glibc available.

Okay, I'm out of arguments. I still don't like it conceptually (I'd prefer the solution in https://github.com/conda-forge/linux-sysroot-feedstock/issues/63) but I can't see it breaking anything. And I have to test, but this probably fixes my problem, so thanks!

[minrk]

But I don't think that can happen either, because openmpi itself still carries the newer glibc dependency, so you won't be able to install it due to an unsolvable dependency. I don't think the downstream package gets a glibc dependency that's not represented in the requirements of the package or its dependencies, though I could have misunderstood something.

[minrk]

@leofang I'm trying to understand your objection to including the flag by default. This flag is in default $LDFLAGS, so it is already used on all shared libraries compiled on conda-forge. This is conda-forge-wide, not specific to openmpi. This change is only making the compiler wrapper more consistent with every link command called on conda-forge.

The only thing that appears to be openmpi-specific here is that CMake's FindMPI ignores LDFLAGS, so without this flag it may not find working mpi. (edit: that was inaccurate)

Including it in the wrapper is also only setting a default, not unconditional, and overridable with OMPI_LDFLAGS (or a later cli flag or $LDFLAGS, which sets the same flag again).

[jakirkham]

Thanks all for the helpful discussion here and weighing potential solutions! 🙏

Wanted to follow up on one point...

FWIW, it's cmake's FindMPI, not openmpi, that is ignoring compiler flags. I think the mistake I made in #158 is to assume that the mpi compiler wrappers should be responsible for respecting $CFLAGS/$LDFLAGS, but that's not right - they should be minimal extensions of $CC/$FC etc. with just enough to find/link mpi.h/libmpi. $CFLAGS/$LDFLAGS should be passed to the compiler wrappers, just like regular compilers, which happens as expected after FindMPI succeeds, but not during for some reason. Sounds plausibly like a CMake bug to me.

If we are able to construct a simple example of this behavior and include it in a new CMake issue, we can work to address it

[minrk]

I think I might be conflating different issues (#158 fixed two issues, one of which fixed FindMPI and it was FCFLAGS-related, not link-related) and getting things wrong. When I run tests, FindMPI definitely does use LDFLAGS.

I've re-read, and the original post by @charlesgwaldman I think is compiling in a user environment with FC=mpifort, (not FindMPI), and I think the issue is that the package gfortran doesn't include compiler activation on linux (it does on mac, I'm not sure why they are inconsistent). If you conda install fortran-compiler (or gfortran_linux-64), LDFLAGS will be defined and everything should work. I've confirmed this in docker with the CMakeLists.txt:

confirming cmake works with $LDFLAGS

```cmake cmake_minimum_required(VERSION 3.28) project(test LANGUAGES Fortran) find_package(MPI) ``` ```bash # docker run --rm -v $PWD:/io -w /io --platform linux/amd64 -it condaforge/miniforge3 conda create -n testbuild cmake make gfortran openmpi=5.0.3=*_104 conda activate testbuild cmake -B build . ``` gives ``` -- The Fortran compiler identification is GNU 13.2.0 -- Detecting Fortran compiler ABI info -- Detecting Fortran compiler ABI info - done -- Check for working Fortran compiler: /opt/conda/envs/testbuild/bin/gfortran - skipped -- Could NOT find MPI_Fortran (missing: MPI_Fortran_WORKS) -- Could NOT find MPI (missing: MPI_Fortran_FOUND) ``` but after adding `fortran-compiler`: ``` conda install fortran-compiler # reactivate env to source activation scripts conda deactivate conda activate testbuild cmake -B build2 . ``` gives: ``` -- The Fortran compiler identification is GNU 12.3.0 -- Detecting Fortran compiler ABI info -- Detecting Fortran compiler ABI info - done -- Check for working Fortran compiler: /opt/conda/envs/testbuild/bin/x86_64-conda-linux-gnu-gfortran - skipped -- Found MPI_Fortran: /opt/conda/envs/testbuild/lib/libmpi_usempif08.so (found version "3.1") -- Found MPI: TRUE (found version "3.1") ```

I think all of the cases where this error is coming up are attributable to $LDFLAGS not being passed, either in package build systems or user environments, and not cmake itself, e.g.:

1. OP is a user environment with gfortran and not gfortran_linux-64, so LDFLAGS is not set
2. @xylar esmf's builds don't appear to use standard $LDFLAGS, instead using ESMF_F90LINKOPTS, set here from $LDFLAGS for mac, but this should be set from $LDFLAGS for all platforms.

Since runtime compilation in user environments is a common and reasonable thing to do, I think it's a valid question to ask: should the mpi compilers work in user environments without the conda-build compiler activation scripts? If so, the answer is either:

1. add run_constrained on sysroot (#160), or
2. add -Wl,--allow-shlib-undefined to the default flags in the compiler wrapper (#159)

(or both)

Note that 2. fixes this issue in all situations, because I think we have learned that building downstream packages with an older sysroot is actually fine, whereas 1. fixes it for user environments, test environments, and cross compiled packages (because openmpi is in the build env, unless we do #161), but the pinning will not affect native builds, where openmpi is only in the host env. Respecting $LDFLAGS, which is a reasonable expectation of conda-forge recipes, solves those cases, though.

run_constrained on sysroot will end up preventing downstream packages from using older sysroot in the build, which I think we've learned actually works fine. In practice, though, once openmpi is on newer glibc, all downstream dependencies might as well update since runtime will require newer glibc due to the dependency so there's no benefit to holding back.