Add QDP-JIT support for JURECA Booster

[martin-ueding]

There is some problem while compiling LLVM with the Intel compiler:

[  0%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/Program.cpp.o
cd /homea/hch02/hch02f/Chroma-2018/build-icc/llvm/lib/Support && /usr/local/software/jurecabooster/Stages/2017b/software/impi/2018.1.163-iccifort-2018.1.163-GCC-5.4.0/bin64/mpiicpc  -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/homea/hch02/hch02f/Chroma-2018/build-icc/llvm/lib/Support -I/homea/hch02/hch02f/Chroma-2018/sources/llvm/lib/Support -I/homea/hch02/hch02f/Chroma-2018/build-icc/llvm/include -I/homea/hch02/hch02f/Chroma-2018/sources/llvm/include  -xMIC-AVX512 -O3 -fopenmp -std=c++11 -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wnon-virtual-dtor -g -fPIC    -fno-exceptions -o CMakeFiles/LLVMSupport.dir/Program.cpp.o -c /homea/hch02/hch02f/Chroma-2018/sources/llvm/lib/Support/Program.cpp
In file included from /homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/ADT/StringMap.h(20),
                 from /homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/Support/Host.h(17),
                 from /homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/ADT/Hashing.h(49),
                 from /homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/ADT/ArrayRef.h(13),
                 from /homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/Support/Program.h(17),
                 from /homea/hch02/hch02f/Chroma-2018/sources/llvm/lib/Support/Program.cpp(14):
/homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/Support/Allocator.h(95): warning #1292: unknown attribute "returns_nonnull"
    LLVM_ATTRIBUTE_RETURNS_NONNULL void *Allocate(size_t Size,
    ^

In file included from /homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/ADT/StringMap.h(20),
                 from /homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/Support/Host.h(17),
                 from /homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/ADT/Hashing.h(49),
                 from /homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/ADT/ArrayRef.h(13),
                 from /homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/Support/Program.h(17),
                 from /homea/hch02/hch02f/Chroma-2018/sources/llvm/lib/Support/Program.cpp(14):
/homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/Support/Allocator.h(211): warning #1292: unknown attribute "returns_nonnull"
    LLVM_ATTRIBUTE_RETURNS_NONNULL LLVM_ATTRIBUTE_RETURNS_NOALIAS void *
    ^

/homea/hch02/hch02f/Chroma-2018/sources/llvm/include/llvm/ADT/StringExtras.h(329): error: "detail" is ambiguous
    return detail::join_impl(Begin, End, Separator, tag());
           ^

compilation aborted for /homea/hch02/hch02f/Chroma-2018/sources/llvm/lib/Support/Program.cpp (code 2)

Without LLVM, no QDP-JIT. So this needs to be fixed.

[martin-ueding]

The above happened with Intel/2018.1.163-GCC-5.4.0.

Then I have tried it with Intel/2017.2.174-GCC-5.4.0 and get an “illegal instruction” during the CMake call, which seems really odd. Presumably it tries to run some code during the configuration stage and cannot do this because we are cross-compiling for KNL.

cmake '-DCMAKE_CXX_FLAGS=-axMIC-AVX512 -O3 -fopenmp -std=c++11' -DCMAKE_CXX_COMPILER=/usr/local/software/jurecabooster/Stages/2017a/software/impi/2017.2.174-iccifort-2017.2.174-GCC-5.4.0/bin64/mpiicpc -DLLVM_ENABLE_TERMINFO=OFF -DCMAKE_C_COMPILER=/usr/local/software/jurecabooster/Stages/2017a/software/impi/2017.2.174-iccifort-2017.2.174-GCC-5.4.0/bin64/mpiicc '-DCMAKE_C_FLAGS=-axMIC-AVX512 -O3 -std=c99 -fopenmp' -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=/homea/hch02/hch02f/Chroma-2018/local-icc -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_ZLIB=OFF -DBUILD_SHARED_LIBS=ON -DLLVM_ENABLE_RTTI=ON /homea/hch02/hch02f/Chroma-2018/sources/llvm
/homea/hch02/hch02f/bootstrap-chroma: line 985: 17372 Illegal instruction     cmake -DCMAKE_CXX_FLAGS="$cxxflags" -DCMAKE_CXX_COMPILER="$(which $cxx_name)" -DLLVM_ENABLE_TERMINFO=OFF -DCMAKE_C_COMPILER="$(which $cc_name)" -DCMAKE_C_FLAGS="$cflags" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX="$prefix" -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_ZLIB=OFF -DBUILD_SHARED_LIBS=ON -DLLVM_ENABLE_RTTI=ON "$sourcedir/$repo"

[kostrzewa]

I think you would need to use clang throughout...

[martin-ueding]

Tom and Jan-Lukas got some bundle of installation scripts for this, Bálint wrote something about using Intel 2016 because he had trouble with the newer releases, but did not specify what. The illegal instruction in CMake can come from JURECA or the code, hard to tell.

I will see whether this can be done with Clang. But given our experience with GCC and ICC on Hazel Hen with QPhiX (40 vs. 60 GFLOP/S), I would fear that the QPhiX performance would be sub-par when we do a Clang-only build. But I can just try to see whether that at least works.

[kostrzewa]

I see. Then it should be done using Intel 16... The earlierst I will find some time to play around with this is after Easter, I'm afraid...

[martin-ueding]

Compiling LLVM with GCC 5.4 has worked, now I can start to work on QDP-JIT. There it seems that CUDA is a hard requirement, hopefully I get an answer to the issue I raised there.