ROCm / HIP

HIP: C++ Heterogeneous-Compute Interface for Portability
https://rocmdocs.amd.com/projects/HIP/
MIT License
3.75k stars 533 forks source link

HIP clang vs. deprecated HCC #1196

Closed tdd11235813 closed 5 months ago

tdd11235813 commented 5 years ago

Can you shed some light on the new compiler workflow on HIP(AMD)? We currently use the deprecated HCC for our HIP codes on the AMD platform (ROCm 2.5). By reading hipcc script, it looks like, the new workflow will involve the /opt/rocm/llvm as default for HIP clang path, which is currently not present on our ROCm 2.5 installation (ubuntu), but it probably will come with new ROCm? Or is it part of the llvm AMDGPU toolchain?

tdd11235813 commented 5 years ago

update, WIP: working on compiling hip-clang via INSTALL.md (please fix AMD-clang link to HIP-clang in TOC). Here is the current install script. It compiles, but could not compile hip code with hip-clang yet. It still uses HCC stuff, but it seems it is intended. I'll update when I know more...

manual install hip-clang (broken)

```bash #!/bin/bash # https://github.com/ROCm-Developer-Tools/HIP/blob/master/INSTALL.md#hip-clang # requirements: # libelf-dev and rocm libs with hcc (atm still required for compiling HIP) #BUILD_TYPE="RelWithDebInfo" BUILD_TYPE="Release" SRC_DIR=$HOME/sources/hipclang LLVM_INSTALL_DIR=/opt/rocm/llvm # make sure you have the privileges HIP_INSTALL_DIR=${LLVM_INSTALL_DIR}/hip # within llvm dir to avoid clash with regular hip installation mkdir -p $SRC_DIR && cd $SRC_DIR # Users need to build LLVM, clang, lld, ROCm device library, and HIP from source # llvm [ -d "llvm/" ] || git clone --single-branch --recursive -b amd-common https://github.com/RadeonOpenCompute/llvm.git llvm cd ${SRC_DIR}/llvm/tools [ -d "clang/" ] || git clone --single-branch --recursive -b amd-common https://github.com/RadeonOpenCompute/clang [ -d "lld/" ] || git clone --single-branch --recursive -b amd-common https://github.com/RadeonOpenCompute/lld # build llvm/clang/lld cd ${SRC_DIR}/llvm if [ ! -d "build/" ]; then mkdir build && cd build cmake .. \ -DLLVM_BUILD_TESTS=OFF \ -DLLVM_PARALLEL_COMPILE_JOBS=12 \ -DLLVM_PARALLEL_LINK_JOBS=12 \ -DLLVM_BUILD_UTILS=OFF \ -DLLVM_TARGETS_TO_BUILD="AMDGPU;X86" \ -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \ -DCMAKE_INSTALL_PREFIX=${LLVM_INSTALL_DIR} \ -DLLVM_ENABLE_TERMINFO=OFF # libncurses issue, https://github.com/android-ndk/ndk/issues/574#issuecomment-373580520 make -j 12 || exit 1 make install || exit 1 fi # build ROCm device library with clang built in last branch cd ${SRC_DIR} [ -d "rocmdevlib/" ] || git clone --single-branch --recursive -b master https://github.com/RadeonOpenCompute/ROCm-Device-Libs.git rocmdevlib cd rocmdevlib if [ ! -d "build/" ]; then mkdir build && cd build CC=${LLVM_INSTALL_DIR}/bin/clang \ cmake .. \ -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \ -DLLVM_DIR=${LLVM_INSTALL_DIR} make -j 12 || exit 1 fi # build HIP (with HCC?) from installed ROCm packages cd $SRC_DIR [ -d "HIP/" ] || git clone --single-branch --recursive -b master https://github.com/ROCm-Developer-Tools/HIP.git cd HIP if [ ! -d "build/" ]; then mkdir -p build && cd build export DEVICE_LIB_PATH=${SRC_DIR}/rocmdevlib/build cmake .. \ -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \ -DHIP_COMPILER=clang \ -DCMAKE_INSTALL_PREFIX=${HIP_INSTALL_DIR} \ -DHSA_PATH=/opt/rocm/hsa \ -DHCC_HOME=/opt/rocm/hcc make -j 12 || exit 1 make install || exit 1 fi # After installation, make sure HIP_PATH is pointed to /where/to/install/hip. # By default HIP looks for HSA in /opt/rocm/hsa (can be overridden by setting HSA_PATH environment variable) # By default HIP is installed into /opt/rocm/hip (can be overridden by setting HIP_PATH environment variable). # By default HIP looks for clang in /opt/rocm/llvm/bin (can be overridden by setting HIP_CLANG_PATH environment variable) # By default HIP looks for device library in /opt/rocm/lib (can be overriden by setting DEVICE_LIB_PATH environment variable). # Optionally, consider adding /opt/rocm/bin to your PATH to make it easier to use the tools. # Optionally, set HIPCC_VERBOSE=7 to output the command line for compilation to make sure clang is used instead of hcc. echo echo "Build was successfull." echo echo "Compile your code with HIP-clang by using the following environment variables" echo echo "export HIP_PATH=${HIP_INSTALL_DIR}" echo "export HIP_CLANG_PATH=${LLVM_INSTALL_DIR}/bin" echo "export DEVICE_LIB_PATH=${SRC_DIR}/rocmdevlib/build" # using source dir for now ```

update: the new hip-clang will not use any HCC at all, so do not waste your time to debug this script. The instructions in the public repos are not consistent at the moment.

davidrohr commented 5 years ago

I tried this recipe, but when I try to compile a hip file, I get:

/usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/a-a5df7e.o: in function `__hip_module_ctor':
a.hip.cpp:(.text+0x706): undefined reference to `__hipRegisterFatBinary'
/usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: a.hip.cpp:(.text+0x739): undefined reference to `__hipRegisterFunction'
/usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/a-a5df7e.o: in function `__hip_module_dtor':
a.hip.cpp:(.text+0x75e): undefined reference to `__hipUnregisterFatBinary'

This is defined in hip_clang.cpp, and this file is also compiled by the above script. Really waiting when hip-clang will be part of a ROCm release.

gargrahul commented 5 years ago

@davidrohr Could you please try with #1331 ?

davidrohr commented 5 years ago

Thanks, the fix works.

davidrohr commented 5 years ago

I found another issue with the hip-clang from this recipe: When the command line invoking hipcc contains -fopenmp, I receive the following errors:

/opt/rocm/llvm/lib/clang/10.0.0/include/__clang_cuda_math_forward_declares.h:53:16: error: static declaration of 'abs' follows non-static declaration
__DEVICE__ int abs(int) __NOEXCEPT;
               ^
/usr/include/stdlib.h:837:12: note: previous declaration is here
extern int abs (int __x) __THROW __attribute__ ((__const__)) __wur;
           ^
In file included from a.hip.cpp:3:
In file included from /opt/rocm/llvm/hip//include/hip/hip_runtime.h:56:
In file included from /opt/rocm/llvm/hip//include/hip/hcc_detail/hip_runtime.h:457:

Please not that I am not compiling for cuda, I have no idea why hipcc pull in the clang cuda headers.

tdd11235813 commented 5 years ago

thanks @davidrohr for pushing this, I also had issues with OpenMP and hcc in the past, not only because of the fishy math includes, and had to disable hipcc and OpenMP. I know it does not help :( (we use HIP for nvcc and hcc backend at the moment, without OpenMP support)

There are several people wondering about this HIP-clang path. It might save a lot of our time if AMD could provide a first HIP-clang version or straightforward guide or at least some information on the current plan & issues, since HCC has been deprecated (have not checked recent updates though).

When an HCC-less HIP-clang will be provided eventually, then it would be now waste of time to debug this HIP-clang compilation where HCC is still used.

tdd11235813 commented 5 years ago

update: the new hip-clang will not use any HCC at all, so do not waste your time to debug this script. The instructions in the public repos are not consistent at the moment.

lamikr commented 4 years ago

Can you give some information which branch of llvm and hip could be used for building the hip-clang version so that I could build and install it by myself?

I have been able to build older hcc based hip and use that for building the tensorflow 2.02 from AMD's tensorflow-upsteam repo's r2.0-rocm-hipclang branch thats using the hip clang version. (same problem with the r2.1 branches)

I have only been able to build the tensorflow if I change the flags on r2.0-rocm-hipclang branch to use the hcc based hip.

ppanchad-amd commented 6 months ago

@lamikr Apologies for the lack of response. Can you please test with latest ROCm 6.1.0 (HIP 6.1)? Thanks!