hipcc fails when cuda is installed in /usr

[sthibaul]

Hello,

When cuda is installed in /usr, hipcc in cuda mode fails to compile anything:

$ echo $CUDA_PATH
$ cat test.hip
void f(void) {
}
$ hipcc -v --x cu -c test.hip -o test.o
#$ _NVVM_BRANCH_=nvvm
#$ _SPACE_= 
#$ _CUDART_=cudart
#$ _HERE_=/usr/lib/nvidia-cuda-toolkit/bin
#$ _THERE_=/usr/lib/nvidia-cuda-toolkit/bin
#$ _TARGET_SIZE_=
#$ _TARGET_DIR_=
#$ _TARGET_SIZE_=64
#$ NVVMIR_LIBRARY_DIR=/usr/lib/nvidia-cuda-toolkit/libdevice
#$ PATH=/usr/lib/nvidia-cuda-toolkit/bin:/usr/bin:/bin:/opt/rocm/hip/bin
#$ LIBRARIES=  -L/usr/lib/x86_64-linux-gnu/stubs -L/usr/lib/x86_64-linux-gnu
#$ gcc -D__CUDA_ARCH__=520 -E -x c++  -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -D__NVCC__  -isystem "/usr/include" -isystem "/net/inria/soft/x86_64/rocm-5.1.0/hip/include"  -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=4 -D__CUDACC_VER_BUILD__=152 -D__CUDA_API_VER_MAJOR__=11 -D__CUDA_API_VER_MINOR__=4 -include "cuda_runtime.h" -m64 "test.hip" -o "/tmp/tmpxft_002df6f5_00000000-7_test.cpp1.ii" 
In file included from /usr/include/crt/math_functions.h:8958,
                 from /usr/include/crt/common_functions.h:303,
                 from /usr/include/cuda_runtime.h:115,
                 from <command-line>:
/usr/include/c++/11/cmath:45:15: fatal error: math.h: Aucun fichier ou dossier de ce type
   45 | #include_next <math.h>
      |               ^~~~~~~~
compilation terminated.
# --error 0x1 --

The -isystem /usr/include is what seems to be spurious, dropping it fixes things, as the attached patch does: patch.txt

This can be easily reproduced by using a debian (buster) or ubuntu image with the nvidia-cuda-toolkit rocm-hip-runtime-dev packages installed and the $CUDA_PATH environment variable set to /usr

Samuel

[cgmb]

This problem also affects hipcc for the AMD platform. It had to be patched for Debian packaging, as HIP will be installed to /usr in that case. As far as I understand, it's never valid to pass -isystem <system include path>.

[cgmb]

There's a number of other install locations that would be equally problematic (e.g., /usr/local) and it would be difficult to extend your patch to handle them all. For that reason, I think it might be better to apply the same fix as was used in https://github.com/llvm/llvm-project/commit/6730b44480fcce18bfbbae0c46719250e9eae425.

@sthibaul, could you perhaps try replacing all uses of isystem with idirafter and giving hipcc a test drive? You can use sed -i.bak 's/isystem/idirafter/g' /opt/rocm/bin/hipcc.pl to quickly apply the change, and revert it with mv /opt/rocm/bin/hipcc.pl{.bak,} when you're done.

[sthibaul]

could you perhaps try replacing all uses of isystem with idirafter

I am getting

€ hipcc -v --x cu -c test.hip  -o test.o
nvcc fatal   : Unknown option '-idirafter'

[cgmb]

Opps. That's certainly not going to work. Thanks for checking!

[ppanchad-amd]

@sthibaul Apologies for the lack of response. Can you please test with latest ROCm 6.0.2 (HIP 6.0.32831)? If resolved, please close ticket. Thanks!

[sthibaul]

I only have ROCm 6.0.0 at hand, but it seems fixed there indeed

[linehill]

I'm experiencing this issue on ROCm 6.1.0.

$ hipcc --version
HIP version: 6.1.40091-a8dbc0c19
AMD clang version 17.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-6.1.0 24103 7db7f5e49612030319346f900c08f474b1f9023a)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-6.1.0/llvm/bin
Configuration file: /opt/rocm-6.1.0/lib/llvm/bin/clang++.cfg

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0

$ CUDA_PATH=/usr HIP_PLATFORM=nvidia hipcc hello.hip -o hello -v
#$ _NVVM_BRANCH_=nvvm
#$ _SPACE_= 
#$ _CUDART_=cudart
#$ _HERE_=/usr/lib/nvidia-cuda-toolkit/bin
#$ _THERE_=/usr/lib/nvidia-cuda-toolkit/bin
#$ _TARGET_SIZE_=
#$ _TARGET_DIR_=
#$ _TARGET_SIZE_=64
#$ NVVMIR_LIBRARY_DIR=/usr/lib/nvidia-cuda-toolkit/libdevice
#$ PATH=/usr/lib/nvidia-cuda-toolkit/bin:/home/linehill/.cargo/bin:/home/linehill/.local/bin:/home/linehill/local/bin:/home/linehill/.cargo/bin:/opt/habanalabs/openmpi-4.1.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
#$ LIBRARIES=  -L/usr/lib/x86_64-linux-gnu/stubs -L/usr/lib/x86_64-linux-gnu
#$ gcc -D__CUDA_ARCH__=520 -D__CUDA_ARCH_LIST__=520 -E -x c++  -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -D__NVCC__  -isystem "/usr/include" -isystem "/opt/rocm-6.1.0/include"  -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=5 -D__CUDACC_VER_BUILD__=119 -D__CUDA_API_VER_MAJOR__=11 -D__CUDA_API_VER_MINOR__=5 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include "cuda_runtime.h" -m64 "hello.hip" -o "/tmp/tmpxft_0000dbf6_00000000-9_hello.cpp1.ii" 
In file included from /usr/include/crt/math_functions.h:9075,
                 from /usr/include/crt/common_functions.h:303,
                 from /usr/include/cuda_runtime.h:115,
                 from <command-line>:
/usr/include/c++/11/cmath:45:15: fatal error: math.h: No such file or directory
   45 | #include_next <math.h>
      |               ^~~~~~~~
compilation terminated.
# --error 0x1 --