search

dftfeDevelopers / dftfe

DFT-FE: Real-space DFT calculations using Finite Elements
https://sites.google.com/umich.edu/dftfe
Other
120 stars 34 forks source link

Trying to build dft-fe with PetSC on CPU's (intel/2023.1 + gcc/9.2.0) #34

Open johannjc opened 10 months ago

johannjc commented 10 months ago

Attempting to build separate real and complex versions - and everything is working up to the final linking. Both the real and complex versions fail with similar linking errors.

I am using

dftfe - release1.0 dealiiCustomizedCUDARelease petsc/3.17.5 intel/2023.1 (with gcc/9.2.0) mpiicc and mpiicpc compilers

I noticed (after getting this far) in the user guide it mentions that intel compilers are not supported - and I'm wondering if this is the issue? Or is there something about using the dealiiCustomizedCUDARelease branch but building without cuda support?

$ make VERBOSE=1 /cm/shared/software/cmake/3.20.0/bin/cmake -S/cm/shared/software/dft-fe/1.0.2/b1/src/dftfe -B/cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real --check-build-system CMakeFiles/Makefile.cmake 0 /cm/shared/software/cmake/3.20.0/bin/cmake -E cmake_progress_start /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real/CMakeFiles /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real//CMakeFiles/progress.marks make -f CMakeFiles/Makefile2 all make[1]: Entering directory /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real' make -f CMakeFiles/dftfem.dir/build.make CMakeFiles/dftfem.dir/depend make[2]: Entering directory/cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real' cd /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real && /cm/shared/software/cmake/3.20.0/bin/cmake -E cmake_depends "Unix Makefiles" /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real/CMakeFiles/ dftfem.dir/DependInfo.cmake --color= make[2]: Leaving directory /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real' make -f CMakeFiles/dftfem.dir/build.make CMakeFiles/dftfem.dir/build make[2]: Entering directory/cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real' make[2]: Nothing to be done for CMakeFiles/dftfem.dir/build'. make[2]: Leaving directory/cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real' [ 95%] Built target dftfem make -f CMakeFiles/dftfe.dir/build.make CMakeFiles/dftfe.dir/depend make[2]: Entering directory /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real' cd /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real && /cm/shared/software/cmake/3.20.0/bin/cmake -E cmake_depends "Unix Makefiles" /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real/CMakeFiles/ dftfe.dir/DependInfo.cmake --color= make[2]: Leaving directory/cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real' make -f CMakeFiles/dftfe.dir/build.make CMakeFiles/dftfe.dir/build make[2]: Entering directory /cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real' [ 97%] Linking CXX executable dftfe /cm/shared/software/cmake/3.20.0/bin/cmake -E cmake_link_script CMakeFiles/dftfe.dir/link.txt --verbose=1 /cm/shared/software/intel/2023.1/mpi/2021.9.0/bin/mpiicpc -O2 -fPIC -fopenmp -shared-intel -qopenmp -rdynamic CMakeFiles/dftfe.dir/src/main.cc.o -o dftfe -Wl,-rpath,/cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real:/cm/shared/software/dealii/20230907/b3/lib:/cm/shared/software/slepc/3.17.2/b1/src/slepc-v3.17.2/arch-ci-linux-intel-mkl-single/lib:/cm/shared/software/petsc/3.17.5/b1/src/petsc-3.17.5/arch-ci-linux-intel-mkl-sing le/lib:/cm/shared/software/lle/petsc-hash-pkgs/b3d0e4/lib: libdftfem.so /cm/shared/software/dealii/20230907/b3/lib/libdeal_II.so.9.3.0-pre /usr/lib64/libz.so -lrt /cm/shared/software/p4est/2.2/b1/lib/libp4est.so /cm/shared/software/p4est/2.2/b1/lib/libsc.so -qmkl-ilp64=parallel -lmkl_scalapack_ilp64 -lmkl_blacs_intelmpi_ilp64 -L/cm/shared/software/intel/2023.1/mkl/2023.1.0/lib/intel64 -Wl,--no-as-needed -lmkl_intel_lp64 -lmkl_gnu_thread -lmk l_core -lgomp -lpthread -lm -ldl /cm/shared/software/slepc/3.17.2/b1/src/slepc-v3.17.2/arch-ci-linux-intel-mkl-single/lib/libslepc.so /cm/shared/software/petsc/3.17.5/b1/src/petsc-3.17.5/arch-ci-linux-intel-mkl-single/lib/libpetsc.so /cm/shared/software/intel/2023.1/mkl/2023.1.0/lib/intel64/libmkl_scalapack_lp64.so /cm/shared/software/intel/2023.1/mkl/2023.1.0/lib/intel64/libmkl_blacs_intelmpi_lp64.so /cm/shared/software/intel/2023.1/mkl/2023 .1.0/lib/intel64/libmkl_intel_lp64.so /cm/shared/software/intel/2023.1/mkl/2023.1.0/lib/intel64/libmkl_core.so /cm/shared/software/intel/2023.1/mkl/2023.1.0/lib/intel64/libmkl_sequential.so /cm/shared/software/lle/petsc-hash-pkgs/b3d0e4/lib/libparmetis.so /cm/shared/software/lle/petsc-hash-pkgs/b3d0e4/lib/libmetis.so /usr/lib64/libX11.so -lifport /cm/shared/software/intel/2023.1/compiler/2023.1.0/linux/compiler/lib/intel64_lin/libifcoremt_pic .a /cm/shared/software/intel/2023.1/compiler/2023.1.0/linux/compiler/lib/intel64_lin/libimf.so /cm/shared/software/intel/2023.1/compiler/2023.1.0/linux/compiler/lib/intel64_lin/libsvml.so -lm /cm/shared/software/intel/2023.1/compiler/2023.1.0/linux/compiler/lib/intel64_lin/libipgo.a /cm/shared/software/intel/2023.1/compiler/2023.1.0/linux/compiler/lib/intel64_lin/libirc.so /cm/shared/software/intel/2023.1/compiler/2023.1.0/linux/compiler/lib/ intel64_lin/libirc_s.a -lquadmath -ldl -lAlglib /cm/shared/software/libxc/6.2.2/b1/lib/libxc.a /usr/lib64/libxml2.so /cm/shared/software/spglib/2.1.0/b1/lib64/libsymspg.so /cm/shared/software/elpa/2021.05.002/b1/lib/libelpa_openmp.so icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message. libdftfem.so: undefined reference tovoid dealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::reinit<std::complex >(dealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host> const&, bool)' libdftfem.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<double>, dealii::MemorySpace::Host>::compress_finish(dealii::VectorOperation::values)' libdftfem.so: undefined reference todealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::Vector()' libdftfem.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<double>, dealii::MemorySpace::Host>::zero_out_ghosts() const' libdftfem.so: undefined reference tovoid dealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::reinit<std::complex >(dealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host> const&, bool)' libdftfem.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<float>, dealii::MemorySpace::Host>::swap(dealii::LinearAlgebra::distributed::Vector<std::complex<float>, dealii::MemorySpace::Host>&)' libdftfem.so: undefined reference todealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::Vector()' libdftfem.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<float>, dealii::MemorySpace::Host>::reinit(dealii::IndexSet const&, dealii::IndexSet const&, int const&)' libdftfem.so: undefined reference todealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::zero_out_ghosts() const' libdftfem.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<double>, dealii::MemorySpace::Host>::update_ghost_values() const' libdftfem.so: undefined reference todealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::compress_start(unsigned int, dealii::VectorOperation::values)' libdftfem.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<float>, dealii::MemorySpace::Host>::update_ghost_values_finish() const' libdftfem.so: undefined reference todealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::update_ghost_values() const' libdftfem.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<double>, dealii::MemorySpace::Host>::update_ghost_values_start(unsigned int) const' libdftfem.so: undefined reference todealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::swap(dealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>&)' libdftfem.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<double>, dealii::MemorySpace::Host>::update_ghost_values_finish() const' libdftfem.so: undefined reference todealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::reinit(dealii::IndexSet const&, dealii::IndexSet const&, int const&)' libdftfem.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<float>, dealii::MemorySpace::Host>::update_ghost_values_start(unsigned int) const' libdftfem.so: undefined reference todealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::compress_finish(dealii::VectorOperation::values)' libdftfem.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<double>, dealii::MemorySpace::Host>::compress_start(unsigned int, dealii::VectorOperation::values)' make[2]: *** [dftfe] Error 1 make[2]: Leaving directory/cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real' make[1]: [CMakeFiles/dftfe.dir/all] Error 2 make[1]: Leaving directory `/cm/shared/software/dft-fe/1.0.2/b1/src/dftfe/build/release/real' make: [all] Error 2

dsambit commented 10 months ago

Thank you for your interest in DFT-FE.

Looking at the error message it is likely because of use of intel compiler. It is not due to building dealiiCustomizedCUDARelease branch without cuda as all the error messages are from dealii::MemorySpace::Host. We have made several CMake cleanups in our main development branch: publicGithubDevelop which are not there in the dftfe - release1.0 branch. Could you please try compiling compiling the publicGithubDevelop branch instead as that could potentially fix the issue? Except elpa/2021.05.002 (we now require elpa/2021.11.001) all the other dependencies which you have already installed can be reused. For your reference, the link to the manual for the development version is here https://github.com/dftfeDevelopers/dftfe/raw/manual/manual-develop.pdf.

johannjc commented 10 months ago

Well - I had to force mpiicpc to use -std=c++17 to avoid /cm/shared/software/dft-fe/1.0.2/b2/src/dftfe/src/linAlg/mixingClass.cc(152): error: expected an identifier for (const auto &[key, value] : d_variableHistoryIn)

and it said I needed elpa > 2022, so I used 2023.05.001.

But with those adjustments, the complex case actually built!

However, the real case is still failing in the linking stage - but now there are only two errors...

[100%] Linking CXX executable dftfe icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message. icpc: command line warning #10121: overriding '-march=native' with '-march=native' libdftfe.so: undefined reference to dealii::LinearAlgebra::distributed::Vector<std::complex<float>, dealii::MemorySpace::Host>::reinit(dealii::IndexSet const&, dealii::IndexSet const&, int const&)' libdftfe.so: undefined reference todealii::LinearAlgebra::distributed::Vector<std::complex, dealii::MemorySpace::Host>::reinit(dealii::IndexSet const&, dealii::IndexSet const&, int const&)' make[2]: [dftfe] Error 1 make[1]: [CMakeFiles/dftfe_exe.dir/all] Error 2 make: *** [all] Error 2

dsambit commented 10 months ago

Good to know that the complex case got built successfully. Is this the release1.0 branch or the publicGithubDevelop branch of DFT-FE? just wanted to confirm.

Regarding the issue with the real case, could you please try not using -march=native' flags as those are meant for GNU compilers?

Overall, our experience with Intel compilers are not great due to somewhat similar compilation issues as you have faced and thus we strongly recommend using GNU compilers if GNU compiler environment already available on your HPC machine. There is no performance degradation in using GNU compilers vs intel compilers and all our code development, testing is conducted using GNU compilers. Is there any particular reason for using intel compilers?

johannjc commented 10 months ago

This is using the publicGithubDevelop branch. I had added -march=native, because without it I was seeing

@.*** real]$ make [ 1%] Building CXX object CMakeFiles/dftfe.dir/src/dftfeWrapper.cc.o icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message. In file included from /cm/shared/software/dealii/20230907/b3/include/deal.II/fe/mapping_q_generic.h(26), from /cm/shared/software/dealii/20230907/b3/include/deal.II/fe/mapping_q1.h(22), from /cm/shared/software/dft-fe/1.0.2/b2/src/dftfe/include/headers.h(48), from /cm/shared/software/dft-fe/1.0.2/b2/src/dftfe/include/constraintMatrixInfo.h(24), from /cm/shared/software/dft-fe/1.0.2/b2/src/dftfe/include/dft.h(21), from /cm/shared/software/dft-fe/1.0.2/b2/src/dftfe/src/dftfeWrapper.cc(43): /cm/shared/software/dealii/20230907/b3/include/deal.II/base/vectorization.h(54): error: #error directive: "Mismatch in vectorization capabilities: AVX was detected during configuration of deal.II and switched on, but it is apparently not available for the file you are trying to compile at the moment. Check compilation flags controlling the instruction set, such as -march=native."

error \

I can rebuild everything with gcc... I just usually default to intel compilers and had missed the warnings in the documentation...

On Fri, Oct 27, 2023 at 12:42 PM Sambit Das @.***> wrote:

Good to know that the complex case got built successfully. Is this the release1.0 branch or the publicGithubDevelop branch?

Regarding the issue with the real case, could you please try not using -march=native' flags as those are meant for GNU compilers?

Overall, our experience with Intel compilers are not great due to somewhat similar compilation issues as you have faced and thus we strongly recommend using GNU compilers if GNU compiler environment already available on your HPC machine. There is no performance degradation in using GNU compilers vs intel compilers and all our code development, testing is conducted using GNU compilers. Is there any particular reason for using intel compilers?

— Reply to this email directly, view it on GitHub https://github.com/dftfeDevelopers/dftfe/issues/34#issuecomment-1783210008, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIMXQ76G2FN7BUXVHL2UBDYBPP6HAVCNFSM6AAAAAA6PYAFX2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBTGIYTAMBQHA . You are receiving this because you authored the thread.Message ID: @.***>