Closed sameershende closed 5 months ago
yxsamliu
commented
2 years ago -mllvm is used to pass generic LLVM options. hipcc uses lld for bitcode linking and LLVM codegen, which is affected by LLVM options, therefore -mllvm options need to be passed to lld.
If you need to pass specific arguments to your plugin, you can use -fplugin-arg- e.g. https://github.com/llvm/llvm-project/blob/main/clang/test/Driver/plugin-driver-args.cpp
sameershende
commented
2 years ago I am unable to pass the -fplugin-arg-
But clang didn't like this argument. Any suggestions? Thanks!
ronlieb
commented
2 years ago @yxsamliu who is an expert on using LLVM plugins and could help with this ?
yxsamliu
commented
2 years ago There is documentation about clang plugin arguments https://clang.llvm.org/docs/ClangPlugins.html
sameershende
commented
2 years ago We discussed the issue. The problem is that the hipcc script needs to be updated to support the -cc1 -load arguments. In ROCm 4.5.2, clang works well with the plugin arguments and loads the plugin, but the hipcc script doesn't work as it only supports -fplugin.
yxsamliu
commented
2 years ago hipcc is a wrapper script for clang driver, not clang -cc1.
If you need to pass an option to clang -cc1, you need to use -Xclang followed by the -cc1 option.
sameershende
commented
2 years ago Thanks for your help. Using -Xclang helped solve the issue:
corona212{shende1}177: hipcc -Xclang -cc1 -Xclang -load -Xclang pwd/lib/PrintFunctionNames.so -Xclang -plugin -Xclang print-fns -Xclang -plugin-arg-print-fns -Xclang help householder3.c -c
hipcc-cmd: "/opt/rocm-4.5.2/llvm/bin/clang" -isystem "/opt/rocm-4.5.2/llvm/lib/clang/13.0.0/include/.." -isystem /opt/rocm-4.5.2/hsa/include -isystem "/opt/rocm-4.5.2/hip/include" -O3 -Xclang -cc1 -Xclang -load -Xclang
I can pass the arguments properly!
sameershende
commented
2 years ago When I use the plugin options, with the older Clang 12 compilers, I could generate a .o file using clang -c, but with hipcc -cc1 -load .. -c, it doesn't generate a .o file. How can I get it to do code generation? The TAU plugin for selective instrumentation needs to insert hooks in the compiled object file for performance instrumentation. So, I do need a .o file at the end of the compilation pass.
This is what I tried:
hipcc -Xclang -cc1 -Xclang -load -Xclang pwd/lib/PrintFunctionNames.so -Xclang -plugin -Xclang print-fns -Xclang -plugin-arg-print-fns -Xclang help householder3.c -c
PrintFunctionNames arg = help
Help for PrintFunctionNames plugin goes here
top-level-decl: "size_t"
top-level-decl: "va_list"
top-level-decl: "gnuc_va_list"
top-level-decl: "u_char"
top-level-decl: "u_short"
top-level-decl: "__u_int"
top-level-decl: "u_long"
…
top-level-decl: "checkUnitary"
top-level-decl: "check"
top-level-decl: "transpose"
top-level-decl: "getsign"
top-level-decl: "normalize"
top-level-decl: "min"
top-level-decl: "copyR"
top-level-decl: "matmul"
top-level-decl: "applyR"
top-level-decl: "applyQ"
top-level-decl: "householder"
ls -lt *.o ls: No match.
So, it launches the plugin properly, passes it arguments, but doesn't generate a .o file. Appreciate any help with this issue. This also affects clang compiler that ships with ROCm 4.5.2.
yxsamliu
commented
2 years ago you need to pass -emit-obj to clang -cc1
sameershende
commented
2 years ago Hi,
While this -emit-obj argument works with -cc1 when no plugin is loaded, it does not work with the plugin. I see:
% hipcc -Xclang -cc1 -Xclang -emit-obj -c foo.c
% ls -l .o
-rw------- 1 shende1 shende1 20760 Feb 4 09:12 foo.o
% rm -f foo.o
% hipcc -Xclang -cc1 -Xclang -emit-obj -c -Xclang -load -Xclang pwd/lib/PrintFunctionNames.so -Xclang -plugin -Xclang print-fns -Xclang -plugin-arg-print-fns -Xclang help foo.c
PrintFunctionNames arg = help
Help for PrintFunctionNames plugin goes here
top-level-decl: "size_t"
top-level-decl: "va_list"
top-level-decl: "gnuc_va_list"
top-level-decl: “u_char"
...
top-level-decl: "copyR"
top-level-decl: "matmul"
top-level-decl: "applyR"
top-level-decl: "applyQ"
top-level-decl: "householder"
% ls -l .o
ls: No match.
Thanks,
- SameerOn Feb 4, 2022, at 7:27 AM, Yaxun (Sam) Liu @.***> wrote:
you need to pass -emit-obj to clang -cc1
— Reply to this email directly, view it on GitHub https://github.com/ROCm-Developer-Tools/HIP/issues/2453#issuecomment-1030091650, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUHIOPV7PACHUNHLDIJNT3UZPV4PANCNFSM5LNQZYTQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you modified the open/close state.
yxsamliu
commented
2 years ago can you try -add-plugin instead of -plugin ?
ppanchad-amd
commented
6 months ago @sameershende Do you still need assistance for this ticket? If not, please close. Thanks!
hipcc script accepts the -fplugin= argument correctly but the arguments intended for the LLVM plugin passed through the -mllvm command line option are sent incorrectly to lld and lld complains about these. Here is an example:
/opt/rocm/hip/bin/hipcc -g -c vectoradd_hip.cpp -g -DPROFILING_ON -DTAU_CLANG -DTAU_DOT_H_LESS_HEADERS -fPIC -DPTHREADS -DTAU_PTHREAD_PRELOAD -DTAU_UNIFY -DTAU_LINUX_TIMERS -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE -DTAU_BFD -DHAVE_GNU_DEMANGLE -DHAVE_TR1_HASH_MAP -DEBS_CLOCK_RES=1 -DTAU_STRSIGNAL_OK -DTAU_UNWIND -DTAU_USE_LIBUNWIND -I/home/users/sameer/tau2/x86_64/libunwind-1.3.1-clang/include -DTAU_TRACK_LD_LOADER -DTAU_USE_TLS -DTAU_NO_FORTRAN -DTAU_ENABLE_ROCM -DTAU_ENABLE_ROCPROFILER -DTAU_GPU -DTAU_GPU -DTAU_ELF_BFD -I/home/users/sameer/tau2/include -g -fplugin=/home/users/sameer/tau2/x86_64/lib/shared-rocm-rocprofiler-clang-pthread/plugins/lib/TAU_Profiling_CXX.so -mllvm -tau-input-file=select.tau -o vectoradd_hip.o
lld: error: -plugin-opt=-: lld: Unknown command line argument '-tau-input-file=select.tau'. Try: '/opt/rocm/llvm/bin/lld --help'
lld: Did you mean '--info-output-file=select.tau'?
clang-13: error: amdgcn-link command failed with exit code 1 (use -v to see invocation)
The -fplugin argument goes through properly, but lld gets passed the -tau-input-file=select.tau option that is intended to be passed to the LLVM plugin but not to lld! hipcc needs to be patched to detect the argument following -mllvm and suppress it as it invokes lld. Here is the output in the verbose mode: hipcc-cmd: "/opt/rocm/llvm/bin/clang" -std=c++11 -isystem "/opt/rocm-4.5.0/llvm/lib/clang/13.0.0/include/.." -isystem /opt/rocm/hsa/include -isystem "/opt/rocm/hip/include" --offload-arch=gfx908 --offload-arch=gfx908 -O3 -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -fhip-new-launch-api -g -c -x hip vectoradd_hip.cpp -g -DPROFILING_ON -DTAU_CLANG -DTAU_DOT_H_LESS_HEADERS -fPIC -DPTHREADS -DTAU_PTHREAD_PRELOAD -DTAU_UNIFY -DTAU_LINUX_TIMERS -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE -DTAU_BFD -DHAVE_GNU_DEMANGLE -DHAVE_TR1_HASH_MAP -DEBS_CLOCK_RES=1 -DTAU_STRSIGNAL_OK -DTAU_UNWIND -DTAU_USE_LIBUNWIND -I/home/users/sameer/tau2/x86_64/libunwind-1.3.1-clang/include -DTAU_TRACK_LD_LOADER -DTAU_USE_TLS -DTAU_NO_FORTRAN -DTAU_ENABLE_ROCM -DTAU_ENABLE_ROCPROFILER -DTAU_GPU -DTAU_GPU -DTAU_ELF_BFD -I/home/users/sameer/tau2/include -g -fplugin=/home/users/sameer/tau2/x86_64/lib/shared-rocm-rocprofiler-clang-pthread/plugins/lib/TAU_Profiling_CXX.so -mllvm -tau-input-file=select.tau -o "vectoradd_hip.o" -v AMD clang version 13.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-4.5.0 21422 e2489b0d7ede612d6586c61728db321047833ed8) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/rocm/llvm/bin Found candidate GCC installation: /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9 Selected GCC installation: /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9 Candidate multilib: .;@m64 Candidate multilib: 32;@m32 Selected multilib: .;@m64 Found HIP installation: /opt/rocm, version 4.4.21401 "/opt/rocm-4.5.0/llvm/bin/clang-13" -cc1 -mllvm --amdhsa-code-object-version=4 -triple amdgcn-amd-amdhsa -aux-triple x86_64-unknown-linux-gnu -emit-obj --mrelax-relocations -disable-free -disable-llvm-verifier -discard-value-names -main-file-name vectoradd_hip.cpp -mrelocation-model pic -pic-level 2 -fhalf-no-semantic-interposition -mframe-pointer=none -fno-rounding-math -aux-target-cpu x86-64 -fcuda-is-device -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_isa_version_908.bc -target-cpu gfx908 -debug-info-kind=limited -dwarf-version=5 -debugger-tuning=gdb -v -resource-dir /opt/rocm-4.5.0/llvm/lib/clang/13.0.0 -internal-isystem /opt/rocm-4.5.0/llvm/lib/clang/13.0.0/include/cuda_wrappers -internal-isystem /opt/rocm/include -include __clang_hip_runtime_wrapper.h -isystem /opt/rocm-4.5.0/llvm/lib/clang/13.0.0/include/.. -isystem /opt/rocm/hsa/include -isystem /opt/rocm/hip/include -D PROFILING_ON -D TAU_CLANG -D TAU_DOT_H_LESS_HEADERS -D PTHREADS -D TAU_PTHREAD_PRELOAD -D TAU_UNIFY -D TAU_LINUX_TIMERS -D TAU_LARGEFILE -D _LARGEFILE64_SOURCE -D TAU_BFD -D HAVE_GNU_DEMANGLE -D HAVE_TR1_HASH_MAP -D EBS_CLOCK_RES=1 -D TAU_STRSIGNAL_OK -D TAU_UNWIND -D TAU_USE_LIBUNWIND -I /home/users/sameer/tau2/x86_64/libunwind-1.3.1-clang/include -D TAU_TRACK_LD_LOADER -D TAU_USE_TLS -D TAU_NO_FORTRAN -D TAU_ENABLE_ROCM -D TAU_ENABLE_ROCPROFILER -D TAU_GPU -D TAU_GPU -D TAU_ELF_BFD -I /home/users/sameer/tau2/include -internal-isystem /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9 -internal-isystem /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/x86_64-redhat-linux -internal-isystem /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/backward -internal-isystem /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9 -internal-isystem /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/x86_64-redhat-linux -internal-isystem /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/backward -internal-isystem /opt/rocm-4.5.0/llvm/lib/clang/13.0.0/include -internal-isystem /usr/local/include -internal-isystem /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../x86_64-redhat-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /opt/rocm-4.5.0/llvm/lib/clang/13.0.0/include -internal-isystem /usr/local/include -internal-isystem /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../x86_64-redhat-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O3 -std=c++11 -fdeprecated-macro -fno-autolink -fdebug-compilation-dir=/home/users/sameer/tau2/examples/gpu/hip/vectorAdd -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -vectorize-loops -vectorize-slp -load /home/users/sameer/tau2/x86_64/lib/shared-rocm-rocprofiler-clang-pthread/plugins/lib/TAU_Profiling_CXX.so -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -mllvm -tau-input-file=select.tau -cuid=f8d1fe08d88b4bc0 -fcuda-allow-variadic-functions -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/vectoradd_hip-gfx908-9912f2.o -x hip vectoradd_hip.cpp clang -cc1 version 13.0.0 based upon LLVM 13.0.0git default target x86_64-unknown-linux-gnu ignoring nonexistent directory "/opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../x86_64-redhat-linux/include" ignoring nonexistent directory "/include" ignoring nonexistent directory "/opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../x86_64-redhat-linux/include" ignoring nonexistent directory "/include" ignoring duplicate directory "/opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9" ignoring duplicate directory "/opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/x86_64-redhat-linux" ignoring duplicate directory "/opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/backward" ignoring duplicate directory "/opt/rocm-4.5.0/llvm/lib/clang/13.0.0/include" ignoring duplicate directory "/usr/local/include" ignoring duplicate directory "/usr/include" ignoring duplicate directory "/usr/local/include" ignoring duplicate directory "/opt/rocm-4.5.0/llvm/lib/clang/13.0.0/include" ignoring duplicate directory "/usr/include"
include "..." search starts here:
include <...> search starts here:
/home/users/sameer/tau2/x86_64/libunwind-1.3.1-clang/include /home/users/sameer/tau2/include /opt/rocm-4.5.0/llvm/lib/clang/13.0.0/include/.. /opt/rocm/hsa/include /opt/rocm/hip/include /opt/rocm-4.5.0/llvm/lib/clang/13.0.0/include/cuda_wrappers /opt/rocm/include /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9 /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/x86_64-redhat-linux /opt/rh/devtoolset-9/root/usr/lib/gcc/x86_64-redhat-linux/9/../../../../include/c++/9/backward /opt/rocm-4.5.0/llvm/lib/clang/13.0.0/include /usr/local/include /usr/include End of search list. "/opt/rocm/llvm/bin/lld" -flavor gnu --no-undefined -shared -plugin-opt=-amdgpu-internalize-symbols -plugin-opt=mcpu=gfx908 -plugin-opt=O3 -plugin-opt=-amdgpu-early-inline-all=true -plugin-opt=-amdgpu-function-calls=false -plugin-opt=-tau-input-file=select.tau -o /tmp/vectoradd_hip-gfx908-7c5922.out /tmp/vectoradd_hip-gfx908-9912f2.o lld: error: -plugin-opt=-: lld: Unknown command line argument '-tau-input-file=select.tau'. Try: '/opt/rocm/llvm/bin/lld --help' lld: Did you mean '--info-output-file=select.tau'? clang-13: error: amdgcn-link command failed with exit code 1 (use -v to see invocation) [sameer@instinct vectorAdd]$
hipcc passed the -plugin-opt=-tau-input-file=select.tau option to lld and lld didn't understand it. This issue was created using http://tau.uoregon.edu/tau.tgz configured using: ./configure -rocm -rocprofiler=/opt/rocm/rocprofiler -bfd=download -unwind=download -cc=clang -c++=hipcc -iowrapper -llvm_src=/opt/rocm/llvm -llvm_cxx=/opt/rocm/llvm/bin/clang++; cd examples/gpu/hip/vectoradd; export PATH=/x86_64/bin:$PATH
export TAU_OPTIONS='-optCompInst -optTauSelectFile=select.tau -optVerbose'; touch select.tau ;
export TAU_MAKEFILE=/include/Makefile; make HIPCC=tau_cxx.sh