search

ROCm / HIPIFY

HIPIFY: Convert CUDA to Portable C++ Code
https://rocm.docs.amd.com/projects/HIPIFY/en/latest/
MIT License
503 stars 70 forks source link

[HIPIFY][Issue]: "fatal error: '__clang_cuda_runtime_wrapper.h' file not found" while generating #1462

Closed Gardene-el closed 4 months ago

Gardene-el commented 5 months ago

Problem Description

I tried to translate some CUDA source file into ROCm code, and did: for file in ./EngineGpuKernels/*.cu; do hipify-clang $file -o "${file%.cu}.cpp" --cuda-path=/opt/cuda ; done (Also for .cuh file)

foreach file i got:

warning: CUDA version is newer than the latest partially supported version 11.8 [-Wunknown-cuda-version]
<built-in>:1:10: fatal error: '__clang_cuda_runtime_wrapper.h' file not found
#include "__clang_cuda_runtime_wrapper.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
hipify-clang: /opt/rocm/lib/llvm/include/clang/Basic/DirectoryEntry.h:142: RefTy &clang::FileMgr::MapEntryOptionalStorage<clang::FileEntryRef>::value() & [RefTy = clang::FileEntryRef]: Assertion `has_value()' failed.
 #0 0x000059a9431d9013 (/opt/rocm/bin/hipify-clang+0x2a2f013)
 #1 0x000059a9431d620f (/opt/rocm/bin/hipify-clang+0x2a2c20f)
 #2 0x000059a9431d635d (/opt/rocm/bin/hipify-clang+0x2a2c35d)
 #3 0x000072b54905a770 (/usr/lib/libc.so.6+0x3c770)
 #4 0x000072b5490ab32c (/usr/lib/libc.so.6+0x8d32c)
 #5 0x000072b54905a6c8 raise (/usr/lib/libc.so.6+0x3c6c8)
 #6 0x000072b5490424b8 abort (/usr/lib/libc.so.6+0x244b8)
 #7 0x000072b5490423dc (/usr/lib/libc.so.6+0x243dc)
 #8 0x000072b549052d46 (/usr/lib/libc.so.6+0x34d46)
 #9 0x000059a941551d00 (/opt/rocm/bin/hipify-clang+0xda7d00)
#10 0x000059a9426b8581 (/opt/rocm/bin/hipify-clang+0x1f0e581)
#11 0x000059a9426c9a05 (/opt/rocm/bin/hipify-clang+0x1f1fa05)
#12 0x000059a9426cb7e9 (/opt/rocm/bin/hipify-clang+0x1f217e9)
#13 0x000059a9426cc4ca (/opt/rocm/bin/hipify-clang+0x1f224ca)
#14 0x000059a942684e87 (/opt/rocm/bin/hipify-clang+0x1edae87)
#15 0x000059a942686e33 (/opt/rocm/bin/hipify-clang+0x1edce33)
#16 0x000059a94270be10 (/opt/rocm/bin/hipify-clang+0x1f61e10)
#17 0x000059a94177858e (/opt/rocm/bin/hipify-clang+0xfce58e)
#18 0x000059a9417758de (/opt/rocm/bin/hipify-clang+0xfcb8de)
#19 0x000059a941551999 (/opt/rocm/bin/hipify-clang+0xda7999)
#20 0x000059a9415e42b9 (/opt/rocm/bin/hipify-clang+0xe3a2b9)
#21 0x000059a9415f9371 (/opt/rocm/bin/hipify-clang+0xe4f371)
#22 0x000059a94177508a (/opt/rocm/bin/hipify-clang+0xfcb08a)
#23 0x000059a94176c161 (/opt/rocm/bin/hipify-clang+0xfc2161)
#24 0x000059a9417701b4 (/opt/rocm/bin/hipify-clang+0xfc61b4)
#25 0x000059a941772fd8 (/opt/rocm/bin/hipify-clang+0xfc8fd8)
#26 0x000059a9417697b1 (/opt/rocm/bin/hipify-clang+0xfbf7b1)
#27 0x000059a941572567 (/opt/rocm/bin/hipify-clang+0xdc8567)
#28 0x000072b549043cd0 (/usr/lib/libc.so.6+0x25cd0)
#29 0x000072b549043d8a __libc_start_main (/usr/lib/libc.so.6+0x25d8a)
#30 0x000059a9414fee45 (/opt/rocm/bin/hipify-clang+0xd54e45)
Aborted (core dumped)

About #703, there is a difference that:

clang --version
clang version 17.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Operating System

Arch Linux

CPU

13th Gen Intel(R) Core(TM) i9-13980HX

ROCm Version

ROCm 6.0.0

ROCm Component

HIPIFY

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

emankov commented 4 months ago

Hello @Gardene-el

  1. Could you please run your hipification command with the -v option and provide a full log for a single source file?

  2. Could you specify the --clang-resource-directory= option to point to the Clang resource directory, which is the parent directory for the include folder that contains __clang_cuda_runtime_wrapper.h and other header files used during the hipification process?

For instance: ./hipify-clang square.cu --cuda-path=/usr/local/cuda-12.3 --clang-resource-directory=/usr/llvm/18.1.5/dist/lib/clang/18

Thanks, Evgeny

Gardene-el commented 4 months ago
  1. Here is a full log for single source file: 
    
>hipify-clang ./CUDA/TOs.cuh -o ./ROCm/TOs.h -v
clang version 17.0.0
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/13.2.1
Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/14.1.1
Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/13.2.1
Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1
Selected GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /opt/cuda, version 
Found HIP installation: /opt/rocm, version 6.0.32831
warning: CUDA version is newer than the latest partially supported version 11.8 [-Wunknown-cuda-version]
clang Invocation:
"/opt/rocm/bin/clang-tool" "-cc1" "-triple" "x86_64-pc-linux-gnu" "-target-sdk-version=11.8" "-aux-triple" "nvptx64-nvidia-cuda" "-fsyntax-only" "-disable-free" "-clear-ast-before-backend" "-main-file-name" "TOs.cuh-49c91c.hip" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mframe-pointer=all" "-fmath-errno" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-debugger-tuning=gdb" "-v" "-fcoverage-compilation-dir=/home/Cr0c0diLe/Documents/alien/source/EngineGpuKernels" "-resource-dir" "/opt/rocm/lib/llvm/lib/clang/17" "-internal-isystem" "/opt/rocm/lib/llvm/lib/clang/17/include/cuda_wrappers" "-include" "__clang_cuda_runtime_wrapper.h" "-I" "/home/Cr0c0diLe/Documents/alien/source/EngineGpuKernels/CUDA" "-I/opt/intel/oneapi/tbb/2021.11/env/../include" "-I/opt/intel/oneapi/mpi/2021.11/include" "-I/opt/intel/oneapi/mkl/2024.0/include" "-I/opt/intel/oneapi/ippcp/2021.9/include" "-I/opt/intel/oneapi/ipp/2021.10/include" "-I/opt/intel/oneapi/dpl/2022.3/include" "-I/opt/intel/oneapi/dpcpp-ct/2024.0/include" "-I/opt/intel/oneapi/dnnl/2024.0/include" "-I/opt/intel/oneapi/dev-utilities/2024.0/include" "-I/opt/intel/oneapi/dal/2024.0/include/dal" "-I/opt/intel/oneapi/compiler/2024.0/opt/oclfpga/include" "-I/opt/intel/oneapi/ccl/2021.11/include" "-internal-isystem" "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1" "-internal-isystem" "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/x86_64-pc-linux-gnu" "-internal-isystem" "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/backward" "-internal-isystem" "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1" "-internal-isystem" "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/x86_64-pc-linux-gnu" "-internal-isystem" "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/backward" "-internal-isystem" "/opt/rocm/lib/llvm/lib/clang/17/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../x86_64-pc-linux-gnu/include" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-internal-isystem" "/opt/rocm/lib/llvm/lib/clang/17/include" "-internal-isystem" "/usr/local/include" "-internal-isystem" "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../x86_64-pc-linux-gnu/include" "-internal-externc-isystem" "/include" "-internal-externc-isystem" "/usr/include" "-internal-isystem" "/opt/cuda/include" "-Wno-pragma-once-outside-header" "-std=c++14" "-fdeprecated-macro" "-fdebug-compilation-dir=/home/Cr0c0diLe/Documents/alien/source/EngineGpuKernels" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fcxx-exceptions" "-fexceptions" "-fcolor-diagnostics" "-internal-isystem" "/opt/rocm/bin/../include/hipify/cuda_wrappers" "-internal-isystem" "/opt/rocm/bin/../include/hipify" "-internal-isystem" "/opt/rocm/bin/include/cuda_wrappers" "-internal-isystem" "/opt/rocm/bin/include" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-x" "cuda" "/tmp/TOs.cuh-49c91c.hip"

clang -cc1 version 17.0.0 based upon LLVM 17.0.0git default target x86_64-pc-linux-gnu ignoring nonexistent directory "/opt/rocm/lib/llvm/lib/clang/17/include/cuda_wrappers" ignoring nonexistent directory "/opt/rocm/lib/llvm/lib/clang/17/include" ignoring nonexistent directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../x86_64-pc-linux-gnu/include" ignoring nonexistent directory "/include" ignoring nonexistent directory "/opt/rocm/lib/llvm/lib/clang/17/include" ignoring nonexistent directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../x86_64-pc-linux-gnu/include" ignoring nonexistent directory "/include" ignoring nonexistent directory "/opt/rocm/bin/../include/hipify/cuda_wrappers" ignoring nonexistent directory "/opt/rocm/bin/../include/hipify" ignoring nonexistent directory "/opt/rocm/bin/include/cuda_wrappers" ignoring nonexistent directory "/opt/rocm/bin/include" ignoring duplicate directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1" ignoring duplicate directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/x86_64-pc-linux-gnu" ignoring duplicate directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/backward" ignoring duplicate directory "/usr/local/include" ignoring duplicate directory "/usr/include"

include "..." search starts here:

include <...> search starts here:

/home/Cr0c0diLe/Documents/alien/source/EngineGpuKernels/CUDA /opt/intel/oneapi/tbb/2021.11/env/../include /opt/intel/oneapi/mpi/2021.11/include /opt/intel/oneapi/mkl/2024.0/include /opt/intel/oneapi/ippcp/2021.9/include /opt/intel/oneapi/ipp/2021.10/include /opt/intel/oneapi/dpl/2022.3/include /opt/intel/oneapi/dpcpp-ct/2024.0/include /opt/intel/oneapi/dnnl/2024.0/include /opt/intel/oneapi/dev-utilities/2024.0/include /opt/intel/oneapi/dal/2024.0/include/dal /opt/intel/oneapi/compiler/2024.0/opt/oclfpga/include /opt/intel/oneapi/ccl/2021.11/include /usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1 /usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/x86_64-pc-linux-gnu /usr/lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/backward /usr/local/include /usr/include /opt/cuda/include End of search list.

:1:10: fatal error: '__clang_cuda_runtime_wrapper.h' file not found #include "__clang_cuda_runtime_wrapper.h" ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ hipify-clang: /opt/rocm/lib/llvm/include/clang/Basic/DirectoryEntry.h:142: RefTy &clang::FileMgr::MapEntryOptionalStorage::value() & [RefTy = clang::FileEntryRef]: Assertion `has_value()' failed. #0 0x00005b6dc6fe8013 (/opt/rocm/bin/hipify-clang+0x2a2f013) #1 0x00005b6dc6fe520f (/opt/rocm/bin/hipify-clang+0x2a2c20f) #2 0x00005b6dc6fe535d (/opt/rocm/bin/hipify-clang+0x2a2c35d) #3 0x000072b66fe50ae0 (/usr/lib/libc.so.6+0x3cae0) #4 0x000072b66fea8e44 (/usr/lib/libc.so.6+0x94e44) #5 0x000072b66fe50a30 raise (/usr/lib/libc.so.6+0x3ca30) #6 0x000072b66fe384c3 abort (/usr/lib/libc.so.6+0x244c3) #7 0x000072b66fe383df (/usr/lib/libc.so.6+0x243df) #8 0x000072b66fe48c67 (/usr/lib/libc.so.6+0x34c67) #9 0x00005b6dc5360d00 (/opt/rocm/bin/hipify-clang+0xda7d00) #10 0x00005b6dc64c7581 (/opt/rocm/bin/hipify-clang+0x1f0e581) #11 0x00005b6dc64d8a05 (/opt/rocm/bin/hipify-clang+0x1f1fa05) #12 0x00005b6dc64da7e9 (/opt/rocm/bin/hipify-clang+0x1f217e9) #13 0x00005b6dc64db4ca (/opt/rocm/bin/hipify-clang+0x1f224ca) #14 0x00005b6dc6493e87 (/opt/rocm/bin/hipify-clang+0x1edae87) #15 0x00005b6dc6495e33 (/opt/rocm/bin/hipify-clang+0x1edce33) #16 0x00005b6dc651ae10 (/opt/rocm/bin/hipify-clang+0x1f61e10) #17 0x00005b6dc558758e (/opt/rocm/bin/hipify-clang+0xfce58e) #18 0x00005b6dc55848de (/opt/rocm/bin/hipify-clang+0xfcb8de) #19 0x00005b6dc5360999 (/opt/rocm/bin/hipify-clang+0xda7999) #20 0x00005b6dc53f32b9 (/opt/rocm/bin/hipify-clang+0xe3a2b9) #21 0x00005b6dc5408371 (/opt/rocm/bin/hipify-clang+0xe4f371) #22 0x00005b6dc558408a (/opt/rocm/bin/hipify-clang+0xfcb08a) #23 0x00005b6dc557b161 (/opt/rocm/bin/hipify-clang+0xfc2161) #24 0x00005b6dc557f1b4 (/opt/rocm/bin/hipify-clang+0xfc61b4) #25 0x00005b6dc5581fd8 (/opt/rocm/bin/hipify-clang+0xfc8fd8) #26 0x00005b6dc55787b1 (/opt/rocm/bin/hipify-clang+0xfbf7b1) #27 0x00005b6dc5381567 (/opt/rocm/bin/hipify-clang+0xdc8567) #28 0x000072b66fe39c88 (/usr/lib/libc.so.6+0x25c88) #29 0x000072b66fe39d4c __libc_start_main (/usr/lib/libc.so.6+0x25d4c) #30 0x00005b6dc530de45 (/opt/rocm/bin/hipify-clang+0xd54e45) Aborted (core dumped) ``` 2. error occured while running instruction with option `--clang-resource-directory=` , here is log: ``` >hipify-clang ./CUDA/TOs.cuh -o ./ROCm/TOs.h --cuda-path=/opt/cuda -v --clang-resource-directory=/usr/bin/clang [HIPIFY] error: hipify-clang: Unknown command line argument '--clang-resource-directory=/usr/bin/clang'. Try: 'hipify-clang --help' hipify-clang: Did you mean '--cuda-gpu-arch=/usr/bin/clang'? Program aborted due to an unhandled Error: hipify-clang: Unknown command line argument '--clang-resource-directory=/usr/bin/clang'. Try: 'hipify-clang --help' hipify-clang: Did you mean '--cuda-gpu-arch=/usr/bin/clang'? #0 0x00005914b0bea013 (/opt/rocm/bin/hipify-clang+0x2a2f013) #1 0x00005914b0be720f (/opt/rocm/bin/hipify-clang+0x2a2c20f) #2 0x00005914b0be735d (/opt/rocm/bin/hipify-clang+0x2a2c35d) #3 0x000079798fa50ae0 (/usr/lib/libc.so.6+0x3cae0) #4 0x000079798faa8e44 (/usr/lib/libc.so.6+0x94e44) #5 0x000079798fa50a30 raise (/usr/lib/libc.so.6+0x3ca30) #6 0x000079798fa384c3 abort (/usr/lib/libc.so.6+0x244c3) #7 0x00005914b0b2d862 (/opt/rocm/bin/hipify-clang+0x2972862) #8 0x00005914aef82857 (/opt/rocm/bin/hipify-clang+0xdc7857) #9 0x000079798fa39c88 (/usr/lib/libc.so.6+0x25c88) #10 0x000079798fa39d4c __libc_start_main (/usr/lib/libc.so.6+0x25d4c) #11 0x00005914aef0fe45 (/opt/rocm/bin/hipify-clang+0xd54e45) Aborted (core dumped) ```
emankov commented 4 months ago

Thanks, @Gardene-el.

  1. It looks like rocm doesn't contain clang resource headers: ignoring nonexistent directory "/opt/rocm/lib/llvm/lib/clang/17/include"
  2. It looks like you are using hipify-clang out of the box, and it is a bit outdated and doesn't contain the support for --clang-resource-directory.

Could you try to locate the __clang_cuda_runtime_wrapper.h file in your system manually?

Gardene-el commented 4 months ago

for 2. Yes, I indeed use the ROCm through AUR, which is a bit outdated that also caused other problem like compile error. I guess that is the source of many problems i met.

The location of __clang_cuda_runtime_wrapper.h file should be /usr/lib/clang/17/include/__clang_cuda_runtime_wrapper.h, and here is whole log of locate __clang_cuda_runtime_wrapper.h :

/home/Cr0c0diLe/.cache/clangd/index/__clang_cuda_runtime_wrapper.h.FE1BEC2CAAE8E39F.idx
/home/Cr0c0diLe/.nuget/packages/cppsharp/1.0.45.22293/contentFiles/any/any/lib/clang/14.0.0/include/__clang_cuda_runtime_wrapper.h
/home/Cr0c0diLe/Documents/UnrealEngine/Engine/Extras/ThirdPartyNotUE/SDKs/HostLinux/Linux_x64/v22_clang-16.0.6-centos7/aarch64-unknown-linux-gnueabi/lib/clang/16/include/__clang_cuda_runtime_wrapper.h
/home/Cr0c0diLe/Documents/UnrealEngine/Engine/Extras/ThirdPartyNotUE/SDKs/HostLinux/Linux_x64/v22_clang-16.0.6-centos7/x86_64-unknown-linux-gnu/lib/clang/16/include/__clang_cuda_runtime_wrapper.h
/opt/ComputeCpp-CE-2.11.0-x86_64-linux-gnu/lib/clang/8.0.0/include/__clang_cuda_runtime_wrapper.h
/opt/intel/oneapi/2024.0/opt/dpct/lib/clang/18/include/__clang_cuda_runtime_wrapper.h
/opt/intel/oneapi/dpcpp-ct/2024.0/opt/dpct/lib/clang/18/include/__clang_cuda_runtime_wrapper.h
/opt/rocm/lib/llvm/lib/clang/17.0.0/include/__clang_cuda_runtime_wrapper.h
/usr/lib/clang/17/include/__clang_cuda_runtime_wrapper.h
/usr/lib/llvm15/lib/clang/15.0.7/include/__clang_cuda_runtime_wrapper.h
/usr/lib/llvm16/lib/clang/16/include/__clang_cuda_runtime_wrapper.h
/usr/lib/zig/include/__clang_cuda_runtime_wrapper.h
emankov commented 4 months ago

The location of __clang_cuda_runtime_wrapper.h file should be /usr/lib/clang/17/include/__clang_cuda_runtime_wrapper.h

Nope, if we talk about ROCm. ROCm's location of __clang_cuda_runtime_wrapper.h is /opt/rocm/lib/llvm/lib/clang/17.0.0/include/

And here, we have a known (and already eliminated) issue with clang versioning. They changed it in the 17th version.

As a workaround solution for your configuration, you can copy /opt/rocm/lib/llvm/lib/clang/17.0.0/include/*.* to /opt/rocm/lib/llvm/lib/clang/17/include/*.* - hipify-clang will work correctly.

A preferred solution is to install the latest ROCm.

Gardene-el commented 4 months ago

The workaround solution works, and since that I'm not going to try preferred solution.

Thanks for your patient help!