/usr/include leaking into conda-build with openmp runtimes enabled.

[DrTodd13]

Comment:

I'm not using the conda-forge recipes (instead basing what I'm doing on the Numba recipe) but I thought this group might have expertise to answer my question. In build.sh, I'm building clang/lld/libunwind and the openmp runtimes as below.

_cmake_config+=(-DLLVM_ENABLE_PROJECTS:STRING="clang;lld;libunwind") _cmake_config+=(-DLLVM_ENABLE_RUNTIMES:STRING=openmp)

I'm doing this in a Centos 7 VM. The build gets into building the runtime and for some of the compilation steps I get the following errors.

[ 12%] Building LLVM bitcode Reduction.cpp-sm_52.bc
cd $SRC_DIR/llvm/build/runtimes/runtimes-bins/openmp/libomptarget/DeviceRTL && $SRC_DIR/llvm/build/bin/clang -S -x c++ -std=c++17 -fvisibility=hidden -O1 -mllvm -openmp-opt-disable -DSHARED_SCRATCHPAD_SIZE=512 -Xclang -emit-llvm-bc -Xclang -aux-triple -Xclang x86_64-unknown-linux-gnu -fopenmp -fopenmp-cuda-mode -Xclang -fopenmp-is-device -I$SRC_DIR/openmp/libomptarget/DeviceRTL/include -I$SRC_DIR/openmp/libomptarget/DeviceRTL/../include -I$SRC_DIR/llvm/include -I$SRC_DIR/llvm/build/include -DOMPTARGET_DEBUG=0 -Xclang -target-cpu -Xclang sm_52 -target nvptx64-nvidia-cuda -Xclang -target-feature -Xclang +ptx61 -D__CUDA_ARCH__=520 $SRC_DIR/openmp/libomptarget/DeviceRTL/src/Reduction.cpp -o Reduction.cpp-sm_52.bc
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/openmp/libomptarget/plugins/common/elf_common/elf_common.cpp:16:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/Object/Binary.h:17:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Triple.h:12:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Twine.h:12:
/home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/SmallVector.h:191:12: error: no template named 'enable_if_t' in namespace 'std'; did you mean 'enable_if'?
      std::enable_if_t<!std::is_same<std::remove_const_t<ItTy>, T *>::value,
      ~~~~~^~~~~~~~~~~
           enable_if
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/type_traits:1766:12: note: 'enable_if' declared here
    struct enable_if
           ^
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/openmp/libomptarget/plugins/common/elf_common/elf_common.cpp:16:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/Object/Binary.h:17:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Triple.h:12:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Twine.h:12:
/home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/SmallVector.h:191:43: error: no template named 'remove_const_t' in namespace 'std'; did you mean 'remove_const'?
      std::enable_if_t<!std::is_same<std::remove_const_t<ItTy>, T *>::value,
                                     ~~~~~^~~~~~~~~~~~~~
                                          remove_const
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/type_traits:1327:12: note: 'remove_const' declared here
    struct remove_const
           ^
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/openmp/libomptarget/plugins/common/elf_common/elf_common.cpp:16:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/Object/Binary.h:17:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Triple.h:12:
In file included from /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/Twine.h:12:
/home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/work/llvm/include/llvm/ADT/SmallVector.h:204:12: error: no template named 'enable_if_t' in namespace 'std'; did you mean 'enable_if'?
      std::enable_if_t<!std::is_same<std::remove_const_t<ItTy>, T *>::value,
      ~~~~~^~~~~~~~~~~
           enable_if
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/type_traits:1766:12: note: 'enable_if' declared here
    struct enable_if
           ^
...

It seems to be trying to get the C++ include files from /usr/include and failing because enable_if_t is not available in the base gcc available on centos 7 (gcc 4.8.5). I don't know why it is trying to use this gcc at all since the conda environment created at the beginning has gcc 7.5 in it as follows:

## Package Plan ##

  environment location: /home/taanders/miniconda3/envs/pyomp39/conda-bld/llvmdev_1689873850502/_build_env

The following NEW packages will be INSTALLED:

    _libgcc_mutex:                        0.1-main
    _openmp_mutex:                        5.1-1_gnu
    _sysroot_linux-64_curr_repodata_hack: 3-haa98f57_10
    binutils_impl_linux-64:               2.35.1-h27ae35d_9
    binutils_linux-64:                    2.35.1-h454624a_30
    bzip2:                                1.0.8-h7b6447c_0
    c-ares:                               1.19.0-h5eee18b_0
    ca-certificates:                      2023.05.30-h06a4308_0
    cmake:                                3.26.4-h96355d8_0
    curl:                                 7.88.1-h5eee18b_1
    expat:                                2.4.9-h6a678d5_0
    gcc_impl_linux-64:                    7.5.0-h7105cf2_17
    gcc_linux-64:                         7.5.0-h8f34230_30
    gdbm:                                 1.18-hd4cb3f1_4
    gettext:                              0.21.0-h39681ba_1
    git:                                  2.40.1-pl5340h36fbf9e_1
    gxx_impl_linux-64:                    7.5.0-h0a5bf11_17
    gxx_linux-64:                         7.5.0-hffc177d_30
    icu:                                  58.2-he6710b0_3
    kernel-headers_linux-64:              3.10.0-h57e8cba_10
    krb5:                                 1.20.1-h143b758_1
    ld_impl_linux-64:                     2.35.1-h7274673_9
    libcurl:                              7.88.1-h251f7ec_1
    libedit:                              3.1.20221030-h5eee18b_0
    libev:                                4.33-h7f8727e_1
    libffi:                               3.4.4-h6a678d5_0
    libgcc-devel_linux-64:                7.5.0-hbbeae57_17
    libgcc-ng:                            11.2.0-h1234567_1
    libgomp:                              11.2.0-h1234567_1
    libnghttp2:                           1.52.0-h2d74bed_1
    libssh2:                              1.10.0-hdbd6064_2
    libstdcxx-devel_linux-64:             7.5.0-hf0c5c8d_17
    libstdcxx-ng:                         11.2.0-h1234567_1
    libuuid:                              1.41.5-h5eee18b_0
    libuv:                                1.44.2-h5eee18b_0
    libxml2:                              2.10.3-hcbfbd50_0
    lz4-c:                                1.9.4-h6a678d5_0
    make:                                 4.2.1-h1bed415_1
    ncurses:                              6.4-h6a678d5_0
    openssl:                              3.0.9-h7f8727e_0
    pcre2:                                10.42-hebb0a14_0
    perl:                                 5.34.0-h5eee18b_2
    pip:                                  23.1.2-py311h06a4308_0
    python:                               3.11.4-h955ad1f_0
    readline:                             8.2-h5eee18b_0
    rhash:                                1.4.3-hdbd6064_0
    setuptools:                           67.8.0-py311h06a4308_0
    sqlite:                               3.41.2-h5eee18b_0
    sysroot_linux-64:                     2.17-h57e8cba_10
    tk:                                   8.6.12-h1ccaba5_0
    tzdata:                               2023c-h04d1e81_0
    wheel:                                0.38.4-py311h06a4308_0
    xz:                                   5.4.2-h5eee18b_0
    zlib:                                 1.2.13-h5eee18b_0
    zstd:                                 1.5.5-hc292b87_0

Any suggestions on how to proceed? Thanks!

[isuruf]

Is it using the just built clang to build openmp?

[DrTodd13]

In the second line of the error report above you can see "$SRC_DIR/llvm/build/bin/clang" as the exe of the build command so, yes, I believe that it is using the just built clang. If I try to build the runtime without including clang as a project to build then it gives me an error so I assume that is a requirement of building the runtime.

[isuruf]

Yeah, you need some patches to clang to automatically find the sysroot. (Or set --sysroot to make clang aware of the conda sysroot)

[isuruf]

Specifically https://github.com/conda-forge/clangdev-feedstock/blob/main/recipe/patches/0002-Fix-sysroot-detection-for-linux.patch and https://github.com/conda-forge/llvmdev-feedstock/blob/main/recipe/build.sh#L22 Or https://github.com/conda-forge/clangdev-feedstock/blob/main/recipe/patches/0003-clang-add-conda-specific-env-var-CONDA_BUILD_SYSROOT.patch

[h-vetinari]

Should we backport those patches to LLVM/clang 15?

[DrTodd13]

@isuruf It seems that for the runtime build portion, when cmake is trying to use the newly built clang to check for the compiler capabilities, that all those checks are failing because of the following:

(/home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690419527432/_build_env) [username@localhost _build_env]$ $SRC_DIR/llvm/build/bin/clang ~/openmp/pthread_test_cmake.c
/usr/bin/ld: cannot find crtbegin.o: No such file or directory
/usr/bin/ld: cannot find -lgcc
/usr/bin/ld: cannot find -lgcc
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

Suggestions? Thanks!

[isuruf]

Did you use my suggestions above?

[isuruf]

Should we backport those patches to LLVM/clang 15?

Don't see why not

[DrTodd13]

Did you use my suggestions above?

Yes, I used the second one.

[isuruf]

CONDA_BUILD_SYSROOT is set in your env?

[DrTodd13]

(/home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690419527432/_build_env) [username@localhost llvmdev_1690419527432]$ echo $CONDA_BUILD_SYSROOT
/home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690419527432/_build_env/x86_64-conda-linux-gnu/sysroot

[DrTodd13]

(/home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690419527432/_build_env) [username@localhost llvmdev_1690419527432]$ find . -name "libgcc*" -print
./work/clang/test/Driver/Inputs/aarch64-linux-gnu-tree/gcc-10/lib/gcc/aarch64-unknown-linux-gnu/10/libgcc.a
./work/clang/test/Driver/Inputs/aarch64-linux-gnu-tree/gcc-7.5.0/lib/gcc/aarch64-unknown-linux-gnu/7.5.0/libgcc.a
./work/clang/test/Driver/Inputs/aarch64-linux-gnu-tree/gcc-9.3.0/lib/gcc/aarch64-unknown-linux-gnu/9.3.0/libgcc.a
./work/clang/test/Driver/Inputs/aarch64-linux-gnu-tree/gcc-9.3.1/lib/gcc/aarch64-unknown-linux-gnu/9.3.1/libgcc.a
./work/clang/test/Driver/Inputs/basic_avr_tree/usr/lib/gcc/avr/5.4.0/libgcc.a
./work/clang/test/Driver/Inputs/basic_avr_tree_2/opt/local/lib/gcc/avr/10.3.0/libgcc.a
./work/clang/test/Driver/Inputs/montavista_i686_tree/usr/lib/gcc/i686-montavista-linux/4.2.0/libgcc.a
./_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/conda-meta/libgcc-ng-11.2.0-h1234567_1.json
./_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/libgcc_s.so
./_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/libgcc_s.so.1
./_build_env/conda-meta/libgcc-devel_linux-64-7.5.0-hbbeae57_17.json
./_build_env/conda-meta/libgcc-ng-11.2.0-h1234567_1.json
./_build_env/x86_64-conda-linux-gnu/sysroot/lib64/libgcc_s.so
./_build_env/x86_64-conda-linux-gnu/sysroot/lib64/libgcc_s.so.1
./_build_env/lib/gcc/x86_64-conda-linux-gnu/7.5.0/libgcc.a
./_build_env/lib/gcc/x86_64-conda-linux-gnu/7.5.0/libgcc_eh.a
./_build_env/lib/libgcc_s.so
./_build_env/lib/libgcc_s.so.1

[isuruf]

Hmm, you might need the other two patches too.

[DrTodd13]

Hmm, you might need the other two patches too.

I thought the same and I tried it. After the rebuild, activating the build environment and rerunning the pthread test program, It changed the linker from /usr/bin/ld to /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/_build_env/bin/x86_64-conda-linux-gnu-ld but otherwise the errors are the same about crtbegin.o and -lgcc not being found.

[h-vetinari]

Sidenote: Please post code & shell output enclosed in triple backticks.

[h-vetinari]

Should we backport those patches to LLVM/clang 15?

Don't see why not

https://github.com/conda-forge/llvmdev-feedstock/pull/232 & https://github.com/conda-forge/clangdev-feedstock/pull/234

[h-vetinari]

Though I note that the errors mention clang-14:

clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

so the backports to the 15.x branch won't have an effect. How are you installing clang @DrTodd13?

[DrTodd13]

Though I note that the errors mention clang-14:

clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

so the backports to the 15.x branch won't have an effect. How are you installing clang @DrTodd13?

Yes indeed, I am backporting to llvm 14.0.6. You can see what I'm doing in my first post where I show a part of build.sh where I am enabling the clang, lld, and libunwind projects along with the openmp runtime. I'm not "installing" clang...it is just being built as part of the llvm build process and then a later part of the build process that is doing the runtime is using the clang that was built earlier in the build.

[DrTodd13]

@isuruf @h-vetinari I found https://discourse.llvm.org/t/cannot-find-lgcc-with-new-clang-worked-with-clang-7/4232/7.

Running clang --verbose on the clang that is being built in the conda build environment finds no candidate gcc implementations.

$SRC_DIR/llvm/build/bin/clang --verbose clang version 14.0.6 (/home/username/miniconda3/envs/pyomp39/conda-bld/git_cache/github.com/Python-for-HPC/llvm-project.git 3f24beb247fa4f527be15ba155d8eff338744c75) Target: x86_64-conda-linux-gnu Thread model: posix InstalledDir: /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/work/llvm/build/bin

[isuruf]

Try

ln -sf  /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/x86_64-conda-linux-gnu /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/work/llvm/build/x86_64-conda-linux-gnu

[DrTodd13]

Try

ln -sf  /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/x86_64-conda-linux-gnu /home/username/miniconda3/envs/pyomp39/conda-bld/llvmdev_1690494537428/work/llvm/build/x86_64-conda-linux-gnu

That directory didn't exist directly under llvmdev_1690494537428 but does exist in the _build_env subdir. I tried doing the ln from that directory to work/llvm/build but it had no effect.

[h-vetinari]

The following NEW packages will be INSTALLED:

From what I can tell you are not using conda-forge packages for the build environment. Can you do

conda config --add channels conda-forge
conda config --set channel_priority strict

and try again with a fresh build environment?

[DrTodd13]

The following NEW packages will be INSTALLED:

From what I can tell you are not using conda-forge packages for the build environment. Can you do

conda config --add channels conda-forge
conda config --set channel_priority strict

and try again with a fresh build environment?

Did a completely clean environment and got everything from conda-forge and it still gives me the same error.

[h-vetinari]

Checking in on this old issue again. AFAIU everything should be fixed for v15+. Is there still a problem @DrTodd13?