Closed sreevalli92 closed 2 years ago
Yes, it does. You need to compile chroma with QDP-JIT, https://github.com/JeffersonLab/qdp-jit, instead of QPD++. And, launch as many processes as GPUs are on each node.
Are there any definitive build instructions available for a QDP-JIT + CHROMA + QMP stack? I have a difficult time selecting the correct combination of CUDA, llvm compiler, CHROMA branch, and QDP-JIT branch that can robustly build on NVIDIA architectures.
Since the main complications for building stems from qdp-jit let me throw together some instructions how to build this.
I just now saw that the devel
branch has CMake support, that's great! I'm building LLVM 13 out of the box and will try to link QDP-JIT against it with CHROMA master. I have an Nc agnostic stack with QUDA support for the inverter.
If the LLVM 13 builds and I can link "out of the box" (or with minor tweaks to the default flags) I can rustle up some CMake code that allows the user to pull LLVM from source and build as a dependency.
Yes, currently you want to use the devel branches from qdp-jit and chroma. Master it is for QMP. CMake is a requirement now, autotools doesn't swing it no more. LLVM 13 is good too. Do "release" build with enabled target "nvptx", no need for clang etc. C++20 compiler is required (GCC 10 or 11) unless you turn off the propagator optimizations in which case a C++14 compiler can build it.
Building LLVM as a dependency sounds intriguing. Definitely curious.
I made some decent headway with LLVM and QMP, but QIO, xpath_reader, and filedb are missing. Must one download and build them manually?
Building For CUDA
QDP++: Configuring System
QDP++: Nc=4
QDP++: Nd=4
QDP++: Ns=4
QDP++: Configuring for Parscalar build
QDP++: Enabling CB2 (4D Checkerboard) Layout
QDP++: Setting Base Precision to 64
QDP++: Setting alignment size to 64
Found LLVM 13.0.1
Using LLVMConfig.cmake in /usr/workspace/howarth1/StealthDM/qdpjit_sandbox/install_llvm/lib/cmake/llvm
CMake Error at CMakeLists.txt:293 (add_subdirectory):
The source directory
/usr/workspace/howarth1/StealthDM/qdpjit_sandbox/qdp-jit/other_libs/xpath_reader
does not contain a CMakeLists.txt file.
CMake Error at CMakeLists.txt:298 (add_subdirectory):
The source directory
/usr/workspace/howarth1/StealthDM/qdpjit_sandbox/qdp-jit/other_libs/filedb
does not contain a CMakeLists.txt file.
CMake Error at CMakeLists.txt:302 (add_subdirectory):
The source directory
/usr/workspace/howarth1/StealthDM/qdpjit_sandbox/qdp-jit/other_libs/qio
does not contain a CMakeLists.txt file.
Configuring incomplete, errors occurred!
See also "/usr/workspace/howarth1/StealthDM/qdpjit_sandbox/build/CMakeFiles/CMakeOutput.log".
See also "/usr/workspace/howarth1/StealthDM/qdpjit_sandbox/build/CMakeFiles/CMakeError.log".
The directories are there? Make sure to clone recursively and also when you switched branches that the submodules got updated (git submodule update --recursive)
Thank @fwinter I always forget about recursive cloning. May I ask why QMP is not also part of the CMake dependency tree?
With my own gcc 10 install and out-of-the-box LLVM I can cleanly install QDPJIT devel
. I'm pretty sure one can add a download and build option for LLVM, but I guess it's cleaner for the user to ensure that they have a sufficient gcc! I encountered some errors when compiling with Nc=4 on the CHROMA side, I can work through those as needed.
OK, very close...
The stack builds all the way to CHROMA executable link time. The error is posted here:
howarth1@lassen708:/usr/WS2/howarth1/StealthDM/qdpjit_sandbox/build_chroma-Nc4$ make
[ 0%] Built target qdp_lapack
[ 91%] Built target chromalib
[ 92%] Linking CXX executable purgaug
/usr/workspace/howarth1/StealthDM/qdpjit_sandbox/install/lib/libjit.a(qdp_llvm.cc.o): In function `QDP::llvm_init_libdevice()':
qdp_llvm.cc:(.text+0x2084): undefined reference to `llvm::parseBitcodeFile(llvm::MemoryBufferRef, llvm::LLVMContext&, llvm::function_ref<llvm::Optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (llvm::StringRef)>)'
/usr/workspace/howarth1/StealthDM/qdpjit_sandbox/install/lib/libjit.a(qdp_llvm.cc.o): In function `QDP::llvm_backend_init_rocm()':
qdp_llvm.cc:(.text+0x2590): undefined reference to `llvm::TargetRegistry::lookupTarget(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
/usr/workspace/howarth1/StealthDM/qdpjit_sandbox/install/lib/libjit.a(qdp_llvm.cc.o): In function `QDP::llvm_backend_init_cuda()':
qdp_llvm.cc:(.text+0x44e0): undefined reference to `llvm::TargetRegistry::lookupTarget(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, llvm::Triple&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
collect2: error: ld returned 1 exit status
make[2]: *** [mainprogs/main/CMakeFiles/purgaug.dir/build.make:279: mainprogs/main/purgaug] Error 1
make[1]: *** [CMakeFiles/Makefile2:377: mainprogs/main/CMakeFiles/purgaug.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
There are three LLVM functions that are missing. They exist in the LLVM (13) source. I'll roll back to LLVM 12 and see if that fixes anything.
Same with LLVM 12. Linking errors are usually fixable with some CMake tweaking, I'll give it some more thought. If you have a QDP-JIT + CHROMA stack script I'll happily take it, but if not I can post my findings here. I'll need to make some minor pushes to QDP-JIT for Nc>3 builds (adding llvm_ne, merging CHROMA master into devel) but nothing major.
Maybe it's nothing but I saw you're using LLVM 13.0.1. (Your output: Found LLVM 13.0.1). The library was developed with version 13.0.0. It's probably nothing, but where did you find that version. I can't see it on llvm.org.
To understand better these linking errors, how about building with 'make VERBOSE=1'.
You need to add llvm_ne? as in 'not equal"? I'm going to try and build with Nc=4. Let me see what happens..
EDIT: With Nc=4 qdpjit builds fine, chroma did not. The reunit routine seems to be Nc=3 specific and needs guarding.
After merging master into development for chroma the reunited issue is resolved. It's probably this addition that required the llvm_ne
line 404 on the commit.
This is the gitlog from my LLVM, pulled from githut and branch release/13.x
howarth1@lassen709:/usr/workspace/howarth1/StealthDM/qdpjit_sandbox/llvm-project$ git log
commit 73daeb3d507f7c8da52a35311ec1799f161ac7a5 (HEAD -> release/13.x, origin/release/13.x)
Author: Artem Belevich <tra@google.com>
Date: Wed Sep 29 15:02:36 2021 -0700
[CUDA] Make sure <string.h> is included with original __THROW defined.
Otherwise we may end up with an inconsistent redeclarations of the standard
library functions if _FORTIFY_SOURCE is in effect.
https://bugs.llvm.org/show_bug.cgi?id=47869
Differential Revision: https://reviews.llvm.org/D110781
(cherry picked from commit 29e00b29f76adb15a51c1ccd6c1fdb6fce5f4d7b)
Also getting errors building the QDP examples. If you grant push rights I can push the version I'm working with.
Hey Dean. Indeed llvm_ne
was missing. It was actually called in a templated function but standard Chroma never instantiated it. I can see how you ran into this. Fixed.
I see you're using typedefs like LatticeColorMatrixFNC. Isn't this exactly what LatticeColorMatrixF is..? In which case I'd suggest to use the latter. Then there's no need to add these.
I see you're checking out LLVM via git pull. I've done this in the past but have switched to using tagged releases. More predictable outcome. I have thrown together some build instructions. Not sure this helps with the linking issue.
There are other routines that are Nc3 specific, like polylp
. From eyeballing this it seems it's silly to restrict it the Nc3. Might just work for general Nc.
So the issue with linking was a PEBKAU type. Lassen has only gcc 8 and I was using custom built gcc 12 (which is doubly odd because I pulled gcc 10 src). By turning off contraction optimisations the stack built and linked with gcc 8 and Nc=4, except for some Nc=3 specific routines that I can either turn on or instantiate manually.
For qdpjit, I placed NC guards around ColorCross and the baryon contractions. I can also quickly write up some CMake that will pull a git tagged LLVM and build with the correct target type for NVIDIA or AMD arch. The same can also be done for QMP. Is that something you'd like in QDP-JIT?
Thank you very much for your help, saved a lot of time!
Oh, the LatticeColorMatrixFNC
and LatticeColorMatrixDNC
are just preprocessor defines that preserve Nc=3 optimisations in CHROMA. If Nc=3 is selected then the templates are instantiated with the optimal Nc=3 args, else they carry the generic QDP_NC.
I want to try CHROMA on multinode multigpu setup.Does it supports and scalable?