Closed ye-luo closed 3 years ago
Yes. In Lassen/Sierra, with clang 11.0.0 and std=c++17 I get to this line which never finishes. It compiles fine with c++14 or with clang-10:
$ time /usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-clang-11.0.0/bin/mpicxx -DADD_ -DH5_USE_16_API -DHAVE_CONFIG_H -DHAVE_LIBBOOST -DHAVE_LIBHDF5 -DUSE_MPI -DUSE_REAL_STRUCT_FACTOR -Drestrict=__restrict__ -I/g/g91/correaa/workspace/qmcpack/src -I/g/g91/correaa/workspace/qmcpack/build/clang11/src -I/g/g91/correaa/workspace/qmcpack/external_codes/boost_multi -I/g/g91/correaa/workspace/qmcpack/external_codes/catch -I/g/g91/correaa/workspace/qmcpack/src/Utilities -I/g/g91/correaa/workspace/qmcpack/external_codes/mpi_wrapper -I/g/g91/correaa/workspace/qmcpack/src/Containers -I/g/g91/correaa/workspace/qmcpack/src/Platforms -I/g/g91/correaa/workspace/qmcpack/src/io -isystem /usr/tcetmp/packages/boost/boost-1.70.0/include -isystem /usr/include/libxml2 --gcc-toolchain=/usr/tce/packages/gcc/gcc-7.3.1/ -std=c++17 -o CMakeFiles/afqmc.dir/AFQMCFactory.cpp.o -c /g/g91/correaa/workspace/qmcpack/src/AFQMC/AFQMCFactory.cpp
For reference, I am using this cmake
line:
$ cmake -DBUILD_AFQMC=1 -DBoost_NO_SYSTEM_PATHS=ON -DBOOST_ROOT=/usr/tcetmp/packages/boost/boost-1.70.0 -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_C_FLAGS="--gcc-toolchain=/usr/tce/packages/gcc/gcc-7.3.1/" \
-DCMAKE_CXX_FLAGS="--gcc-toolchain=/usr/tce/packages/gcc/gcc-7.3.1/" \
-DCUDA_HOST_COMPILER=/usr/tce/packages/clang/clang-11.0.0/bin/clang++ \
@mmorale3 I created this PR to experiment in Lassen
Clang 11.x with C++14 also doesn't work.
Ye, can you give us the exact version of the clang11 compiler you are using?
On Bora, ubuntu 18.04, both cases work clang version 11.0.0 (https://github.com/llvm/llvm-project.git 176249bd6732a8044d457092ed932768724a6f06) clang version 11.0.1 (https://github.com/llvm/llvm-project.git 43ff75f2c3feef64f9d73328230d34dac8832a91) Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
On Epyc, ubuntu 20.04, both cases fail. clang 11.0.0 and 11.0.1 commits are the same as above. Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
It seems an issue with the underlying GCC version.
Hmm. AFQMC is enabled in some of the nitrogen and sulfur builds with clang 11.0.1. e.g. https://cdash.qmcpack.org/CDash/buildSummary.php?buildid=187416 (clang11 full real precision build). Build is successful.
The clangdev ("13.0.0") builds currently don't use MPI for simplicity on these machines so AFQMC is not tested. However it could be added or I could check if of interest. (I still think it very unfortunate to require MPI for anything. Besides the dependency, it complicates diagnoses and tool usage.)
@prckent what is the underlying gcc version of your clang 11.0.1? clang -v
Underlying gcc is 10.2.0 for nightly clang 11.0.1 and clangdev builds.
This is just to confirm that I can compile with clang11 and gcc-toolchain 7.3.1:
Here it is how I was able to compile with clang11.
$ git clone git@github.com:QMCPACK/qmcpack.git develop
$ mkdir -p develop/build/clang11
$ cd develop/build/clang11
$ head ~/people/mmorale/build.sh
$ ~/people/mmorale/build.sh
compiles well, tests ctest -R afqmc
pass.
Typical compilation line (issued by make
) looks like this (clang11, cuda 11.2, --gcc-toolchain=/usr/tce/packages/gcc/gcc-7.3.1/):
cd /usr/gapps/afqmc/QMCPACK/branches/develop/build/clang11/src/QMCTools && /usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-clang-11.0.0/bin/mpicxx -DADD_ -DENABLE_PHDF5 -DH5_USE_16_API -DHAVE_CONFIG_H -DHAVE_LIBBOOST -DHAVE_LIBHDF5 -DUSE_REAL_STRUCT_FACTOR -Drestrict=__restrict__ -I/usr/tce/packages/cuda/cuda-11.2.0-beta/include -I/usr/gapps/afqmc/QMCPACK/branches/develop/src -I/usr/gapps/afqmc/QMCPACK/branches/develop/build/clang11/src -I/usr/gapps/afqmc/QMCPACK/branches/develop/external_codes/boost_multi -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/QMCDrivers -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/QMCHamiltonians -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/QMCWaveFunctions -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/Particle -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/Utilities -I/usr/gapps/afqmc/QMCPACK/branches/develop/external_codes/mpi_wrapper -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/Containers -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/Platforms -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/io -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/Estimators -isystem /usr/tcetmp/packages/boost/boost-1.70.0/include -isystem /usr/gapps/afqmc/libs/Ray/hdf5-1.8.20/include -isystem /usr/include/libxml2 --gcc-toolchain=/usr/tce/packages/gcc/gcc-7.3.1/ -fopenmp -fstrict-aliasing -D__forceinline=inline -Wvla -Wall -Wno-unused-variable -Wno-overloaded-virtual -Wno-unused-private-field -Wno-unused-local-typedef -Wno-unknown-pragmas -Wmisleading-indentation -mcpu=native -O3 -DNDEBUG -fomit-frame-pointer -ffast-math -O3 -DNDEBUG -std=c++14 -o CMakeFiles/fstool.dir/QMCFiniteSize/SkParserHDF5.cpp.o -c /usr/gapps/afqmc/QMCPACK/branches/develop/src/QMCTools/QMCFiniteSize/SkParserHDF5.cpp
The build.sh
script has:
#!/bin/tcsh
module load clang/11.0.0
setenv CXX mpicxx
setenv CC mpicc
module load cuda/11.2.0-beta
setenv HDF5_ROOT /usr/gapps/afqmc/libs/Ray/hdf5-1.8.20/
setenv BOOST_ROOT /usr/tcetmp/packages/boost/boost-1.70.0/
#setenv XLLIBDIR /usr/tce/packages/xl/xl-beta-2017.11.28/alllibs
set NCCL_HOME = "/usr/gapps/afqmc/libs/NCCL/cuda/targets/ppc64le-linux/"
cmake -DQMC_INCLUDE="${CUDA_HOME}/include" \
-DQMC_EXTRA_LIBS="-I${CUDA_HOME}/include -L${CUDA_HOME}/lib64 -Wl,-rpath=${CUDA_HOME} -lcusolver -lcurand -lcusparse -lcublas -lcudart -L/usr/lib64/atlas/ -lsatlas -Wl,-rpath=/usr/lib64/atlas/" \
-DBoost_NO_SYSTEM_PATHS=ON \
-DCMAKE_C_COMPILER="mpicc" \
-DCMAKE_CXX_COMPILER="mpicxx" \
-DCMAKE_C_FLAGS="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1/" \
-DCMAKE_CXX_FLAGS="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1/" \
-DCUDA_HOST_COMPILER=/usr/tce/packages/clang/clang-10.0.0/bin/clang++ \
-DCMAKE_SKIP_BUILD_RPATH=false -DENABLE_TIMERS=1 -DENABLE_CUDA=1 -DCUDA_ARCH="sm_70"\
-DBUILD_AFQMC=1 -DQMC_COMPLEX=1 -DBUILD_LMYENGINE_INTERFACE=0 \
-DQMC_MIXED_PRECISION=1 -DCMAKE_BUILD_TYPE=Release \
-DENABLE_STACKTRACE=1 \
-DMPIEXEC=jsrun -DMPIEXEC_NUMPROC_FLAG="-n" -DMPIEXEC_PREFLAGS=";" ../../;
make VERBOSE=1 -j 20
Can somebody help me reproduce this problem, I was able to compile with clang 11.0.0
, nvcc 11.2
, gcc-lib 8.3.1
? @mmorale3 @ye-luo
module load clang/11.0.0
...
-DCMAKE_C_FLAGS="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1/" \
-DCMAKE_CXX_FLAGS="--gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1/" \
make
lines are like this:
cd /usr/gapps/afqmc/QMCPACK/branches/develop/build/clang11/src/QMCTools/tests && /usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-clang-11.0.0/bin/mpicxx -DADD_ -DENABLE_PHDF5 -DH5_USE_16_API -DHAVE_CONFIG_H -DHAVE_LIBBOOST -DHAVE_LIBHDF5 -DUSE_REAL_STRUCT_FACTOR -Drestrict=__restrict__ -I/usr/tce/packages/cuda/cuda-11.2.0-beta/include -I/usr/gapps/afqmc/QMCPACK/branches/develop/src -I/usr/gapps/afqmc/QMCPACK/branches/develop/build/clang11/src -I/usr/gapps/afqmc/QMCPACK/branches/develop/external_codes/boost_multi -I/usr/gapps/afqmc/QMCPACK/branches/develop/external_codes/catch -I/usr/gapps/afqmc/QMCPACK/branches/develop/external_codes/mpi_wrapper -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/QMCDrivers -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/QMCHamiltonians -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/QMCWaveFunctions -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/Particle -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/Utilities -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/Containers -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/Platforms -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/io -I/usr/gapps/afqmc/QMCPACK/branches/develop/src/Estimators -isystem /usr/tcetmp/packages/boost/boost-1.70.0/include -isystem /usr/gapps/afqmc/libs/Ray/hdf5-1.8.20/include -isystem /usr/include/libxml2 --gcc-toolchain=/usr/tce/packages/gcc/gcc-8.3.1/ -fopenmp -fstrict-aliasing -D__forceinline=inline -Wvla -Wall -Wno-unused-variable -Wno-overloaded-virtual -Wno-unused-private-field -Wno-unused-local-typedef -Wno-unknown-pragmas -Wmisleading-indentation -mcpu=native -O3 -DNDEBUG -fomit-frame-pointer -ffast-math -O3 -DNDEBUG -std=c++14 -o CMakeFiles/test_tools.dir/test_qmcfstool.cpp.o -c /usr/gapps/afqmc/QMCPACK/branches/develop/src/QMCTools/tests/test_qmcfstool.cpp
The problem seems to be gcc9, which we don't have in LLNL.
@correaa This issue is exactly Clang 11 using gcc 9 toolchian. On my machine, I just checked Clang >=11 with gcc 7.5(spack), 8.4(spack), 9.2(spack), 9.3(OS), 10.2(spack). both 9.2 and 9.3 caused problems. No need of CUDA, the issue appears even in the CPU only build.
We could try to narrow down further, but is there a reasonable chance of a workaround? The situation is not ideal, but the number of people using Clang 11 is small, and they are likely the kinds of people who can get gcc 10.
If it is not easy to workaround, we can block certain combinations of flags in CMake.
I am trying to reproduce the error locally:
This works:
$ OMPI_CXX="clang++-10" cmake -DCMAKE_CXX_COMPILER="mpicxx" -DBUILD_AFQMC=1 ../..
$ OMPI_CXX="clang++-10" make VERBOSE=1 -j 10
Interestingly, this emits a cmake
error:
$ OMPI_CXX="clang++-10" cmake -DCMAKE_CXX_COMPILER="mpicxx" -DBUILD_AFQMC=1 ../.. -DCMAKE_C_FLAGS="--gcc-toolchain=/usr/lib/gcc/x86_64-linux-gnu/9/" -DCMAKE_CXX_FLAGS="--gcc-toolchain=/usr/lib/gcc/x86_64-linux-gnu/9/"
...
CMake Error at CMake/GNUCompilers.cmake:94 (MESSAGE):
Your system and GNU compiler are using glibc 2.22 or 2.23 which contains a
buggy libmvec. This results in crashes. Workaround needed. Alternatively
upgrade or use another compiler.
...
Do you know if this is the right way to change the toolchain in a standard system?
@ye-luo
what is the output of apt list gcc
and what is your cmake
line for this case you described earlier in the thread?
On Epyc, ubuntu 20.04, both cases fail. clang 11.0.0 and 11.0.1 commits are the same as above. Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
yeluo@epyc-server:~/opt/openmp-target/hands-on/tests/math$ apt list gcc
Listing... Done
gcc/focal,now 4:9.3.0-1ubuntu2 amd64 [installed]
gcc/focal 4:9.3.0-1ubuntu2 i386
cmake line
export OMPI_CC=clang
export OMPI_CXX=clang++
cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DBUILD_AFQMC=ON ..
Ok , I see the same thing in my Ubuntu 20.04 and it works (compiles).
If I do:
export OMPI_CC=clang-11
export OMPI_CXX=clang++-11
cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DBUILD_AFQMC=ON ../..
I get:
cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DBUILD_AFQMC=ON ../..
-- The C compiler identification is Clang 11.0.0
-- The CXX compiler identification is Clang 11.0.0
-- Check for working C compiler: /usr/bin/mpicc
-- Check for working C compiler: /usr/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/mpicxx
-- Check for working CXX compiler: /usr/bin/mpicxx -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/bin/python3.8 (found version "3.8.5") found components: Interpreter
-- Unable to import PySCF python module. PySCF tests will not be run.
-- QMC_SYMLINK_TEST_FILES = ON. Using symbolic links for large test files may cause test failures if the build is installed on a separate filesystem from the source. For example, Titan at OLCF.
-- CMAKE_BUILD_TYPE is RELEASE
-- defining the float point precision
Base precision = double
Full precision = double
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for dlfcn.h
-- Looking for dlfcn.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for memory.h
-- Looking for memory.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/time.h
-- Looking for sys/time.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Found Git: /usr/bin/git (found version "2.25.1")
-- Trying to figure out .....
-- Compiler: Clang
-- libstdc++/C++ compiler version compatibility check failed
CMake Error at CMake/Testlibstdc++.cmake:39 (message):
Change Dir:
/home/correaa/prj/qmcpack_develop/build/clang10/CMakeFiles/CMakeTmp
Run Build Command(s):/usr/bin/make cmTC_db003/fast && /usr/bin/make -f
CMakeFiles/cmTC_db003.dir/build.make CMakeFiles/cmTC_db003.dir/build
make[1]: Entering directory
'/home/correaa/prj/qmcpack_develop/build/clang10/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_db003.dir/try_libstdcxx.cpp.o
/usr/bin/mpicxx -fopenmp -fstrict-aliasing -D__forceinline=inline -Wvla
-Wall -Wno-unused-variable -Wno-overloaded-virtual
-Wno-unused-private-field -Wno-unused-local-typedef -Wno-unknown-pragmas
-Wmisleading-indentation -march=native -std=c++14 -o
CMakeFiles/cmTC_db003.dir/try_libstdcxx.cpp.o -c
/home/correaa/prj/qmcpack_develop/build/clang10/CMakeFiles/CMakeTmp/try_libstdcxx.cpp
Linking CXX executable cmTC_db003
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_db003.dir/link.txt
--verbose=1
/usr/bin/mpicxx -fopenmp -fstrict-aliasing -D__forceinline=inline -Wvla
-Wall -Wno-unused-variable -Wno-overloaded-virtual
-Wno-unused-private-field -Wno-unused-local-typedef -Wno-unknown-pragmas
-Wmisleading-indentation -march=native
CMakeFiles/cmTC_db003.dir/try_libstdcxx.cpp.o -o cmTC_db003
/usr/bin/ld: cannot find -lomp
clang: error: linker command failed with exit code 1 (use -v to see
invocation)
make[1]: *** [CMakeFiles/cmTC_db003.dir/build.make:87: cmTC_db003] Error 1
make[1]: Leaving directory
'/home/correaa/prj/qmcpack_develop/build/clang10/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_db003/fast] Error 2
Call Stack (most recent call first):
CMakeLists.txt:405 (include)
-- Configuring incomplete, errors occurred!
See also "/home/correaa/prj/qmcpack_develop/build/clang10/CMakeFiles/CMakeOutput.log".
@mmorale3 @ye-luo @prckent
I still can't reproduce the problem.
Installed Ubuntu 20.04 in a virtual machine, also installed:
$ sudo apt install clang-11 libblas-dev liblapack-dev libhdf5-dev zlibg1-dev libboost-dev -libfftw3-dev
clang-11 uses gcc-9 toolchain
$ clang++-11 -v
Ubuntu clang version 11.0.0-2~ubuntu20.04.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
regarding gcc
$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
configured qmcpack with:
export OMPI_CC=clang-11
export OMPI_CXX=clang++-11
cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DBUILD_AFQMC=ON ..
(see cmake output at the end)
compiled successfully with
make test_afqmc_numerics
also compiled all
make -j 2
I am starting to think that this is a defect of Spack that installs incompatible subversions of both libraries. Could that be the case?
Cmake ouptut:
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/mpicc
-- Check for working C compiler: /usr/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/mpicxx
-- Check for working CXX compiler: /usr/bin/mpicxx -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/bin/python3.8 (found version "3.8.5") found components: Interpreter
-- Unable to import PySCF python module. PySCF tests will not be run.
-- QMC_SYMLINK_TEST_FILES = ON. Using symbolic links for large test files may cause test failures if the build is installed on a separate filesystem from the source. For example, Titan at OLCF.
-- CMAKE_BUILD_TYPE is RELEASE
-- defining the float point precision
Base precision = double
Full precision = double
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for dlfcn.h
-- Looking for dlfcn.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for memory.h
-- Looking for memory.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/time.h
-- Looking for sys/time.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Found Git: /usr/bin/git (found version "2.25.1")
-- Trying to figure out .....
-- Compiler: GNU
-- libstdc++/C++ compiler version compatibility check pass
-- C++14 standard library supported
-- Found MPI_CXX: /usr/bin/mpicxx (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: CXX
-- MPI runner MPIEXEC_EXECUTABLE : /usr/bin/mpiexec
-- MPIEXEC_NUMPROC_FLAG : -n
-- MPIEXEC_PREFLAGS :
-- Tests run as : /usr/bin/mpiexec -n NUM_PROCS EXECUTABLE
-- MPI is enabled
-- Trying to find LAPACK from Intel MKL
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Could NOT find BLAS (missing: BLAS_LIBRARIES)
-- LAPACK requires BLAS
-- A library with LAPACK API not found. Please specify library location.
-- Intel MKL library files not found via FindLAPACK.
-- Trying to find alternative LAPACK libraries
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- Found BLAS: /usr/lib/x86_64-linux-gnu/libblas.so
-- Looking for cheev_
-- Looking for cheev_ - not found
-- Looking for cheev_
-- Looking for cheev_ - found
-- A library with LAPACK API found.
-- LAPACK linker flags:
-- LAPACK libraries: /usr/lib/x86_64-linux-gnu/liblapack.so;/usr/lib/x86_64-linux-gnu/libblas.so
CMake Warning at CMakeLists.txt:544 (MESSAGE):
AFQMC - MKL not found, using simple sparse matrix routines. Link with MKL
sparse libraries for better performance.
-- No usable vector math library detected.
-- SINCOS_INCLUDE : cmath
-- Performing Test HAVE_SINCOS
-- Performing Test HAVE_SINCOS - Success
-- FFTW_INCLUDE_DIR=/usr/include
-- FFTW_LIBRARIES=/usr/lib/x86_64-linux-gnu/libfftw3.so
-- Found FFTW
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
-- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so (found version "2.9.10")
-- Linking dynamic HDF5 library
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.4") found components: C
-- Serial HDF5 library found
-- Using HDF5 non-scalable serial I/O code paths
CMake Warning at CMakeLists.txt:646 (MESSAGE):
MPI builds may have potential performance loss not using parallel HDF5!
-- Found Boost: /usr/include (found suitable version "1.71.0", minimum required is "1.61.0")
-- Setting Boost_INCLUDE_DIRS=/usr/include
-- CUDA disabled
-- CUDA NVTX APIs disabled
-- VTune ittnotify APIs disabled
Project C_FLAGS: -std=c99 -fopenmp -finline-limit=1000 -fstrict-aliasing -funroll-all-loops -Wno-deprecated -Werror=vla -march=native -O3 -DNDEBUG -fomit-frame-pointer -ffast-math
Project CXX_FLAGS: -fopenmp -finline-limit=1000 -fstrict-aliasing -funroll-all-loops -D__forceinline=inline -Wno-deprecated -Wvla -Wcomment -Wmisleading-indentation -Wmaybe-uninitialized -Wuninitialized -Wreorder -Wno-unknown-pragmas -Wno-sign-compare -march=native -O3 -DNDEBUG -fomit-frame-pointer -ffast-math
Project INCLUDE_DIRECTORIES: /home/correaa/qmcpack/src;/home/correaa/qmcpack/build/src
Project EXE_LINKER_FLAGS:
Project SHARED_LINKER_FLAGS:
-- Ready to build qmcpack
sed supports -E
Git branch: develop
Git commit hash: f6ccf120081c43d7817dc17eb84851b7c6be0450
Building AFQMC performance executable
-- Creating integration tests for QMCPACK
Set QMC_VERBOSE_CONFIGURATION=1 for details of which tests will be enabled.
QE_BIN is not set. Tests of pw2qmcpack and QE/PWSCF will not be performed.
Adding system tests for QMCPACK
Missing python module numpy, not adding test afqmc_workflow
Missing python module h5py, not adding test afqmc_workflow
Missing python module mpi4py, not adding test afqmc_workflow
Missing python module scipy, not adding test afqmc_workflow
Missing python module pyscf, not adding test afqmc_workflow
Missing python module numpy, not adding test pyscf_workflow
Missing python module h5py, not adding test pyscf_workflow
Missing python module pyscf, not adding test pyscf_workflow
Missing python module PyscfToQmcpack, not adding test pyscf_workflow
Missing python module PyscfToQmcpack_Spline, not adding test pyscf_workflow
Skipping converter tests with HDF output because h5diff was not found
Adding estimator tests for QMCPACK
Missing python module numpy, not adding test estimator-skinetic
Missing python module h5py, not adding test estimator-skinetic
Missing python module numpy, not adding test estimator-latdev
Missing python module pandas, not adding test estimator-latdev
Missing python module h5py, not adding test estimator-latdev
Missing python module numpy, not adding test estimator-latdev-free
Missing python module pandas, not adding test estimator-latdev-free
Missing python module h5py, not adding test estimator-latdev-free
Missing python module numpy, not adding test estimator-sofk
Missing python module pandas, not adding test estimator-sofk
Missing python module h5py, not adding test estimator-sofk
Adding I/O tests for QMCPACK
QMC_DATA is not set. Performance tests will be skipped.
Missing python module numpy, not adding test diamond-estimator
Missing python module h5py, not adding test diamond-estimator
Missing python module numpy, not adding test diamond-estimator
Missing python module h5py, not adding test diamond-estimator
Missing python module numpy, not adding test diamond-estimator
Missing python module h5py, not adding test diamond-estimator
Missing python module numpy, not adding test diamond-traces
Missing python module h5py, not adding test diamond-traces
Adding example tests for QMCPACK
Skipping Nexus tests because numpy is not present in python installation
-- Configuring done
-- Generating done
-- Build files have been written to: /home/correaa/qmcpack/build
your cmake output (first two lines) shows you are using gcc.
Are you using MPICH instead of OpenMPI? then probably MPICH_CC and MPICH_CXX
shoot! ubuntu installs mpich by default. Installing openmpi
Good news (or bad), I was able to reproduce the error:
$ mpicxx -v
Ubuntu clang version 11.0.0-2~ubuntu20.04.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
cd /home/correaa/qmcpack/build/src/AFQMC/Numerics/tests && /usr/bin/mpicxx -DADD_ -DH5_USE_16_API -DHAVE_CONFIG_H -DHAVE_LIBBOOST -DHAVE_LIBHDF5 -DUSE_REAL_STRUCT_FACTOR -Drestrict=__restrict__ -I/home/correaa/qmcpack/src -I/home/correaa/qmcpack/build/src -I/home/correaa/qmcpack/external_codes/boost_multi -I/home/correaa/qmcpack/external_codes/catch -I/home/correaa/qmcpack/external_codes/mpi_wrapper -I/home/correaa/qmcpack/src/Utilities -I/home/correaa/qmcpack/src/Containers -I/home/correaa/qmcpack/src/Platforms -I/home/correaa/qmcpack/src/io -isystem /usr/include/hdf5/serial -isystem /usr/include/libxml2 -fopenmp -fstrict-aliasing -D__forceinline=inline -Wvla -Wall -Wno-unused-variable -Wno-overloaded-virtual -Wno-unused-private-field -Wno-unused-local-typedef -Wno-unknown-pragmas -Wmisleading-indentation -march=native -O3 -DNDEBUG -fomit-frame-pointer -ffast-math -std=c++14 -o CMakeFiles/test_afqmc_numerics.dir/test_ma_blas_extensions.cpp.o -c /home/correaa/qmcpack/src/AFQMC/Numerics/tests/test_ma_blas_extensions.cpp
In file included from /home/correaa/qmcpack/src/AFQMC/Numerics/tests/test_ma_blas_extensions.cpp:14:
In file included from /home/correaa/qmcpack/external_codes/catch/catch.hpp:356:
In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/string:41:
In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/allocator.h:46:
In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9/bits/c++allocator.h:33:
/usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/ext/new_allocator.h:146:8: error: calling a protected constructor of class 'boost::multi::basic_array<double, 1, double *, boost::multi::layout_t<1, long>>'
_Up(std::forward<_Args>(__args)...)))
^
...
Another datapoint. In current Debian images clang-11 has a dependency on gcc-10. Perhaps due to these kind of incompatibilities. So the combination gcc-9 and clang-11 is less frequent than it seem.
I fixed multi wrt to the bug in clang11+gcc9 but I am not sure if I did the git subtree add
properly.
https://github.com/QMCPACK/qmcpack/pull/2968
Please test in the reference system. This only contains changes in multi, that are reflected in the main repository.
Is this confirmed fixed after the multi updates?
Good question, I have an Ubuntu 20.04 VM with this configuration and it seems to work.
Note: Ultimately I solved all the main problems by simplifying multi rather than making it more complicated so I am confident it was a good fix. The modifications I pushed for the core of AFQMC were stylistic mostly.
I think this can be closed now, it should be fixed with https://github.com/QMCPACK/qmcpack/pull/3023
correaa@correaa-Standard-PC-i440FX-PIIX-1996:~/qmcpack/build$ rm ../build/* -rf
correaa@correaa-Standard-PC-i440FX-PIIX-1996:~/qmcpack/build$ export OMPI_CC=clang-11
correaa@correaa-Standard-PC-i440FX-PIIX-1996:~/qmcpack/build$ export OMPI_CXX=clang++-11
correaa@correaa-Standard-PC-i440FX-PIIX-1996:~/qmcpack/build$ mpicxx -v
Ubuntu clang version 11.0.0-2~ubuntu20.04.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
correaa@correaa-Standard-PC-i440FX-PIIX-1996:~/qmcpack/build$ cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DBUILD_AFQMC=ON ..
-- The C compiler identification is Clang 11.0.0
-- The CXX compiler identification is Clang 11.0.0
-- Check for working C compiler: /usr/bin/mpicc
-- Check for working C compiler: /usr/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/mpicxx
-- Check for working CXX compiler: /usr/bin/mpicxx -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- QMC_SYMLINK_TEST_FILES = ON. Using symbolic links for large test files may cause test failures if the build is installed on a separate filesystem from the source.
-- Found Python3: /usr/bin/python3.8 (found version "3.8.5") found components: Interpreter
-- Unable to import PySCF python module. PySCF tests will not be run.
-- Did not find a patched Quantum ESPRESSO (QE) distribution with pw2qmcpack.x. QE tests will not be run.
-- CMAKE_BUILD_TYPE is RELEASE
-- defining the float point precision
Base precision = double
Full precision = double
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for dlfcn.h
-- Looking for dlfcn.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for memory.h
-- Looking for memory.h - found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/time.h
-- Looking for sys/time.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for unistd.h
-- Looking for unistd.h - found
-- Found Git: /usr/bin/git (found version "2.25.1")
-- Trying to figure out .....
-- Compiler: Clang
-- libstdc++/C++ compiler version compatibility check pass
-- C++14 standard library supported
-- Setting QMC_SIMD_ALIGNMENT to 32
-- Found MPI_CXX: /usr/bin/mpicxx (found version "3.1")
-- Found MPI: TRUE (found version "3.1") found components: CXX
-- Detected Open MPI. Setting bind-to options for thread affinity in MPIEXEC_PREFLAGS.
-- MPI runner MPIEXEC_EXECUTABLE : /usr/bin/mpiexec
-- MPIEXEC_NUMPROC_FLAG : -n
-- MPIEXEC_PREFLAGS : --bind-to;none
-- Tests run as : /usr/bin/mpiexec -n NUM_PROCS --bind-to none EXECUTABLE
-- MPI is enabled
-- Trying to find LAPACK from Intel MKL
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Could NOT find BLAS (missing: BLAS_LIBRARIES)
-- LAPACK requires BLAS
-- A library with LAPACK API not found. Please specify library location.
-- Intel MKL library files not found via FindLAPACK.
-- Trying to find alternative LAPACK libraries
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- Found BLAS: /usr/lib/x86_64-linux-gnu/libblas.so
-- Looking for cheev_
-- Looking for cheev_ - not found
-- Looking for cheev_
-- Looking for cheev_ - found
-- A library with LAPACK API found.
-- LAPACK linker flags:
-- LAPACK libraries: /usr/lib/x86_64-linux-gnu/liblapack.so;/usr/lib/x86_64-linux-gnu/libblas.so
CMake Warning at CMakeLists.txt:574 (MESSAGE):
AFQMC - MKL not found, using simple sparse matrix routines. Link with MKL
sparse libraries for better performance.
-- No usable vector math library detected.
-- SINCOS_INCLUDE : cmath
-- Performing Test HAVE_SINCOS
-- Performing Test HAVE_SINCOS - Success
-- FFTW_INCLUDE_DIR=/usr/include
-- FFTW_LIBRARIES=/usr/lib/x86_64-linux-gnu/libfftw3.so
-- Found FFTW
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
-- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so (found version "2.9.10")
-- Linking dynamic HDF5 library
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.so;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libsz.so;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/libm.so (found version "1.10.4") found components: C
-- Serial HDF5 library found
-- Using HDF5 non-scalable serial I/O code paths
CMake Warning at CMakeLists.txt:676 (MESSAGE):
MPI builds may have performance loss by not using parallel HDF5! (Safe to
ignore for workstation builds).
-- Found Boost: /usr/include (found suitable version "1.71.0", minimum required is "1.61.0")
-- Setting Boost_INCLUDE_DIRS=/usr/include
-- CUDA disabled
-- CUDA NVTX APIs disabled
-- VTune ittnotify APIs disabled
Project C_FLAGS: -std=c99 -fopenmp -fstrict-aliasing -Werror=vla -march=native -O3 -DNDEBUG -fomit-frame-pointer -ffast-math
Project CXX_FLAGS: -fopenmp -fstrict-aliasing -D__forceinline=inline -Wvla -Wall -Wno-unused-variable -Wno-overloaded-virtual -Wno-unused-private-field -Wno-unused-local-typedef -Wno-unknown-pragmas -Wmisleading-indentation -march=native -O3 -DNDEBUG -fomit-frame-pointer -ffast-math
Project INCLUDE_DIRECTORIES: /home/correaa/qmcpack/src;/home/correaa/qmcpack/build/src
Project EXE_LINKER_FLAGS:
Project SHARED_LINKER_FLAGS:
-- Ready to build qmcpack
sed supports -E
Git branch: develop
Git commit hash: d3382fbdd4b64448bbffdbf5f01a31b0589f6af4
Building AFQMC performance executable
-- Adding integration tests for QMCPACK
Set QMC_VERBOSE_CONFIGURATION=1 for details of which tests will be enabled.
Adding system tests for QMCPACK
Missing python module numpy, not adding test afqmc_workflow
Missing python module h5py, not adding test afqmc_workflow
Missing python module mpi4py, not adding test afqmc_workflow
Missing python module scipy, not adding test afqmc_workflow
Missing python module pyscf, not adding test afqmc_workflow
Skipping converter tests with HDF output because h5diff was not found
Adding estimator tests for QMCPACK
Missing python module numpy, not adding test estimator-skinetic
Missing python module h5py, not adding test estimator-skinetic
Missing python module numpy, not adding test estimator-latdev
Missing python module pandas, not adding test estimator-latdev
Missing python module h5py, not adding test estimator-latdev
Missing python module numpy, not adding test estimator-latdev-free
Missing python module pandas, not adding test estimator-latdev-free
Missing python module h5py, not adding test estimator-latdev-free
Missing python module numpy, not adding test estimator-sofk
Missing python module pandas, not adding test estimator-sofk
Missing python module h5py, not adding test estimator-sofk
Adding I/O tests for QMCPACK
QMC_DATA is not set. Performance tests will be skipped.
Missing python module numpy, not adding test diamond-estimator
Missing python module h5py, not adding test diamond-estimator
Missing python module numpy, not adding test diamond-estimator
Missing python module h5py, not adding test diamond-estimator
Missing python module numpy, not adding test diamond-estimator
Missing python module h5py, not adding test diamond-estimator
Missing python module numpy, not adding test diamond-traces
Missing python module h5py, not adding test diamond-traces
Adding example tests for QMCPACK
Skipping Nexus tests because numpy is not present in python installation
-- Configuring done
-- Generating done
-- Build files have been written to: /home/correaa/qmcpack/build
correaa@correaa-Standard-PC-i440FX-PIIX-1996:~/qmcpack/build$ make
...
correaa@correaa-Standard-PC-i440FX-PIIX-1996:~/qmcpack/build$ ctest -R ppconvert --output-on-failure
Test project /home/correaa/qmcpack/build
Start 23: ppconvert_runs
1/2 Test #23: ppconvert_runs ................... Passed 3.04 sec
Start 24: ppconvert_o_diff
2/2 Test #24: ppconvert_o_diff ................. Passed 0.24 sec
correaa@correaa-Standard-PC-i440FX-PIIX-1996:~/qmcpack/build$ ctest -R afqmc --output-on-failure
Test project /home/correaa/qmcpack/build
Start 14: deterministic-unit_test_afqmc_matrix
1/9 Test #14: deterministic-unit_test_afqmc_matrix .......................... Passed 0.48 sec
Start 15: deterministic-unit_test_afqmc_numerics
2/9 Test #15: deterministic-unit_test_afqmc_numerics ........................ Passed 0.33 sec
Start 16: deterministic-unit_test_afqmc_slaterdeterminantoperations
3/9 Test #16: deterministic-unit_test_afqmc_slaterdeterminantoperations ..... Passed 0.31 sec
Start 17: deterministic-unit_test_afqmc_walkers
4/9 Test #17: deterministic-unit_test_afqmc_walkers ......................... Passed 0.30 sec
Start 18: deterministic-unit_test_afqmc_hamiltonians_ham_chol
5/9 Test #18: deterministic-unit_test_afqmc_hamiltonians_ham_chol ........... Passed 0.32 sec
Start 19: deterministic-unit_test_afqmc_wfn_factory_ham_chol_wfn_rhf
6/9 Test #19: deterministic-unit_test_afqmc_wfn_factory_ham_chol_wfn_rhf .... Passed 0.33 sec
Start 20: deterministic-unit_test_afqmc_phmsd
7/9 Test #20: deterministic-unit_test_afqmc_phmsd ........................... Passed 0.31 sec
Start 21: deterministic-unit_test_afqmc_prop_factory_ham_chol_wfn_rhf
8/9 Test #21: deterministic-unit_test_afqmc_prop_factory_ham_chol_wfn_rhf ... Passed 0.37 sec
Start 22: deterministic-unit_test_afqmc_estimators_ham_chol_wfn_rhf
9/9 Test #22: deterministic-unit_test_afqmc_estimators_ham_chol_wfn_rhf ..... Passed 0.32 sec
100% tests passed, 0 tests failed out of 9
Label Time Summary:
afqmc = 3.07 sec*proc (9 tests)
deterministic = 3.07 sec*proc (9 tests)
quality_unknown = 3.07 sec*proc (9 tests)
unit = 3.07 sec*proc (9 tests)
Total Test time (real) = 3.18 sec
My clang builds are happy ;) @correaa thank you so much to get this fixed and we can move on to release.
Great!
Thanks for your patience to you @ye-luo and @mmorale3
I don't expect this situation to repeat, I now have set up automatic test that always test the new version of multi with QMCPACK. So, even if Multi subtree is not updated often I will be a step ahead.
Also, I tagged the current version of Multi as "v0.75".
Clang 11 and 12 RC report error.
release blocker