include path of hipcc

[VinInn]

I'm confused about the actual origin of the c++ headers for hipcc (I assumed to be our gcc distribution. Instead it seems to come from a local installation

I'm in

[innocent@patatrack03 binary64]$ echo $CMSSW_BASE/
/data/user/innocent/CMSSW_14_1_0/
[innocent@patatrack03 binary64]$ which hipcc
/cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0/external/el8_amd64_gcc12/bin/hipcc
[innocent@patatrack03 binary64]$ hipcc --version
HIP version: 5.6.31062-73ed8adfd
AMD clang version 16.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-5.6.1 23332 4f9bb99d78a4d8d9770be38b91ebd004ea4d2a3a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/bin

If I compile something with it

AMD clang version 16.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-5.6.1 23332 4f9bb99d78a4d8d9770be38b91ebd004ea4d2a3a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/bin
Found candidate GCC installation: /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12
Selected GCC installation: /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/cuda/12.4.1-fc5cb0e72dba64b6abbf00089f3a044c, version
Found HIP installation: /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035, version 5.6.31062
 "/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/bin/clang-16" -cc1 -triple amdgcn-amd-amdhsa -aux-triple x86_64-unknown-linux-gnu -emit-obj -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name oneComplex_hip.cpp -mrelocation-model pic -pic-level 1 -fhalf-no-semantic-interposition -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility=hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/amdgcn/bitcode/oclc_isa_version_900.bc -mlink-builtin-bitcode /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/amdgcn/bitcode/oclc_abi_version_500.bc -target-cpu gfx900 -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -resource-dir /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/lib/clang/16.0.0 -internal-isystem /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/lib/clang/16.0.0/include/cuda_wrappers -idirafter /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/include -include __clang_hip_runtime_wrapper.h -I /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/bin/../include -isystem /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/include -internal-isystem /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12 -internal-isystem /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/x86_64-redhat-linux -internal-isystem /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/backward -internal-isystem /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12 -internal-isystem /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/x86_64-redhat-linux -internal-isystem /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/backward -internal-isystem /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/bin/../include -internal-isystem /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/lib/clang/16.0.0/include -internal-isystem /usr/local/include -internal-isystem /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../x86_64-redhat-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/bin/../include -internal-isystem /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/lib/clang/16.0.0/include -internal-isystem /usr/local/include -internal-isystem /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../x86_64-redhat-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O3 -fdeprecated-macro -fno-autolink -fdebug-compilation-dir=/data/user/innocent/math_accuracy/binary64 -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -vectorize-loops -vectorize-slp -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -cuid=7461e2c05db1f837 -fcuda-allow-variadic-functions -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/oneComplex_hip-ff346f/oneComplex_hip-gfx900.o -x hip oneComplex_hip.cpp
clang -cc1 version 16.0.0 based upon LLVM 16.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../x86_64-redhat-linux/include"
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../x86_64-redhat-linux/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12"
ignoring duplicate directory "/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/x86_64-redhat-linux"
ignoring duplicate directory "/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/backward"
ignoring duplicate directory "/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/bin/../include"
  as it is a non-system directory that duplicates a system directory
ignoring duplicate directory "/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/bin/../include"
ignoring duplicate directory "/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/lib/clang/16.0.0/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/lib/clang/16.0.0/include"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/include"
#include "..." search starts here:
#include <...> search starts here:
 /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/include
 /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/lib/clang/16.0.0/include/cuda_wrappers
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/x86_64-redhat-linux
 /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/backward
 /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/bin/../include
 /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/lib/clang/16.0.0/include
 /usr/local/include
 /usr/include
End of search list.

so it seems to take gcc header files from /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/ which actually does exist

ls /opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12
algorithm  cassert    ciso646    condition_variable  cstdio    debug         filesystem        iostream  memory           pstl              shared_mutex     stdlib.h      tr1            valarray
any        ccomplex   climits    coroutine           cstdlib   decimal       forward_list      istream   memory_resource  queue             source_location  stop_token    tr2            variant
array      cctype     clocale    csetjmp             cstring   deque         fstream           iterator  mutex            random            span             streambuf     tuple          vector
atomic     cerrno     cmath      csignal             ctgmath   exception     functional        latch     new              ranges            spanstream       string        type_traits    version
backward   cfenv      codecvt    cstdalign           ctime     execution     future            limits    numbers          ratio             sstream          string_view   typeindex      x86_64-redhat-linux
barrier    cfloat     compare    cstdarg             cuchar    expected      initializer_list  list      numeric          regex             stack            syncstream    typeinfo
bit        charconv   complex    cstdbool            cwchar    experimental  iomanip           locale    optional         scoped_allocator  stacktrace       system_error  unordered_map
bits       chrono     complex.h  cstddef             cwctype   ext           ios               map       ostream          semaphore         stdatomic.h      tgmath.h      unordered_set
bitset     cinttypes  concepts   cstdint             cxxabi.h  fenv.h        iosfwd            math.h    parallel         set               stdexcept        thread        utility

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @VinInn.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[VinInn]

indeed (added a little #warning in the local file)

[innocent@patatrack03 binary64]$ hipcc -O3 oneComplex_hip.cpp -fopenmp
In file included from <built-in>:1:
In file included from /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/lib/clang/16.0.0/include/__clang_hip_runtime_wrapper.h:125:
In file included from /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/rocm/5.6.1-afadfea30394e14bf2becdbebd346035/llvm/lib/clang/16.0.0/include/cuda_wrappers/complex:82:
/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/complex:39:2: warning: This is complex from local installation [-W#warnings]
#warning This is complex from local installation

[VinInn]

@fwyzard FYI

[smuzaffar]

@VinInn , we pass --gcc-toolchain=$(COMPILER_PATH) to hipcc (as hipcc uses llvm on backend) (https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_14_2_X/master/scram-tools.file/tools/rocm/rocm.xml#L18 ) when used within cmssw. I think llvm 13 and earlier versions were able to pick gcc headers from COMPILER_PATH env variable but with llvm 14 and above we had to explicitly set --gcc-toolchain ( https://github.com/cms-sw/cmsdist/pull/8201 ) . I had opened an issue about this https://github.com/llvm/llvm-project/issues/56343 .