search

mfem / mfem

Lightweight, general, scalable C++ library for finite element methods
http://mfem.org
BSD 3-Clause "New" or "Revised" License
1.71k stars 498 forks source link

Update FindPETSc cmake module #4232

Closed chapman39 closed 7 months ago

chapman39 commented 7 months ago

Located in config/cmake/modules/FindPETSc.cmake, petsc's cmake module is 8 years old at the time of writing this. As a result, modern versions of petsc (e.g. v3.20.4) are unable to be built with MFEM if using the cmake build system. This is affecting the Serac project, as we optionally build MFEM as a submodule using cmake.

The following error output is from an attempt to build MFEM with petsc (version 3.20.4) enabled:

CMake Error at smith/serac/mfem/config/cmake/modules/FindPETSc.cmake:147 (message):
  The pair
  PETSC_DIR=/usr/WS1/topopt/libs/lido/toss_4_x86_64_ib/2024_02_16_16_37_47/linux-rhel8-ivybridge/clang-14.0.6/petsc-3.20.4-iwtouyzn3b2zdjpofsc6n637ae52lsbr
  PETSC_ARCH=arch-linux2-c-debug do not specify a valid PETSc installation
Call Stack (most recent call first):
  smith/serac/mfem/CMakeLists.txt:254 (find_package)

CMake Error at /usr/tce/backend/installations/linux-rhel8-x86_64/gcc-10.3.1/cmake-3.23.1-mdfqd2l7c33zg7xcvqizwz25vqmp7jfw/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  PETSc could not be found.  Be sure to set PETSC_DIR and PETSC_ARCH.
  (missing: PETSC_INCLUDES PETSC_EXECUTABLE_RUNS)
Call Stack (most recent call first):
  /usr/tce/backend/installations/linux-rhel8-x86_64/gcc-10.3.1/cmake-3.23.1-mdfqd2l7c33zg7xcvqizwz25vqmp7jfw/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  smith/serac/mfem/config/cmake/modules/FindPETSc.cmake:349 (find_package_handle_standard_args)
  smith/serac/mfem/CMakeLists.txt:254 (find_package)

Here is a recently updated petsc module used by Serac, but it has not been tested on older versions of petsc or within MFEM: https://github.com/LLNL/serac/blob/develop/cmake/thirdparty/FindPETSc.cmake

Related Serac issue: https://github.com/LLNL/serac/issues/1082

lmolin3 commented 7 months ago

Hi @chapman39 , I had a similar issue recently. Adding the following to the cmake config file solved it for me:

if (MFEM_USE_PETSC)
   set(PETSC_EXECUTABLE_RUNS "ON")  
endif()

I'm using Petsc v3.20.2 in my MFEM Docker image; the code compiles and some examples using petsc seem to run fine.

chapman39 commented 7 months ago

I tried adding the following but I got /usr/bin/ld: cannot find -lpetsc.

        if (MFEM_USE_PETSC)
            set(PETSC_EXECUTABLE_RUNS "ON" CACHE BOOL "")
            set(PETSC_ARCH "" CACHE STRING "")
            if(SLEPC_FOUND)
                set(SLEPC_ARCH "" CACHE STRING "")
            endif()
        endif()
v-dobrev commented 7 months ago

Did you try setting PETSC_DIR and PETSC_ARCH as the error message suggests? If you did, what were the errors in that case, if any?

Edit: for PETSc installation through Spack, you may need to set PETSC_ARCH to be empty.

v-dobrev commented 7 months ago

I tried adding the following but I got /usr/bin/ld: cannot find -lpetsc.

        if (MFEM_USE_PETSC)
            set(PETSC_EXECUTABLE_RUNS "ON" CACHE BOOL "")
            set(PETSC_ARCH "" CACHE STRING "")
            if(SLEPC_FOUND)
                set(SLEPC_ARCH "" CACHE STRING "")
            endif()
        endif()

I'm not sure where you added this code. I was thinking about just setting -DPETSC_DIR=<petsc-dir> -DPETSC_ARCH=<petsc-arch> on the CMake config command line.

Regarding the error /usr/bin/ld: cannot find -lpetsc, can you try to build in verbose mode and post the command that generates this message?

chapman39 commented 7 months ago

Did you try setting PETSC_DIR and PETSC_ARCH as the error message suggests? If you did, what were the errors in that case, if any?

Edit: for PETSc installation through Spack, you may need to set PETSC_ARCH to be empty.

Yeah. Petsc was installed through Spack, so I set the PETSC_ARCH to an empty string. PETSC_DIR is set earlier as well.

I'm not sure where you added this code.

This is being called just before calling add_subdirectory to mfem's source in Serac's setup third party cmake file. Normally we don't do this, but sometimes we "codevelop" with mfem, in case serac users need to change something in mfem's source.

can you try to build in verbose mode and post the command that generates this message?

Yes.

[100%] Linking CXX executable ../../tests/mfem_array_std_algo cd /usr/WS2/meemee/serac/repo/build1/src/tests && /usr/tce/backend/installations/linux-rhel8-x86_64/gcc-10.3.1/cmake-3.23.1-mdfqd2l7c33zg7xcvqizwz25vqmp7jfw/bin/cmake -E cmake_link_script CMakeFiles/mfem_array_std_algo.dir/link.txt --verbose=1 /usr/tce/packages/gcc/gcc-10.3.1/bin/g++ -Wall -Wextra -Werror -Wshadow -Wdouble-promotion -Wconversion -Wundef -Wnull-dereference -Wold-style-cast -O3 -DNDEBUG -rdynamic -Wl,-rpath -Wl,/usr/tce/packages/mvapich2/mvapich2-2.3.7-gcc-10.3.1/lib -fopenmp CMakeFiles/mfem_array_std_algo.dir/mfem_array_std_algo.cpp.o -o ../../tests/mfem_array_std_algo -L/usr/tce/packages/mvapich2/mvapich2-2.3.7-gcc-10.3.1/lib -Wl,-rpath,/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/serac-develop-ndjnfsbzzcdi5gp5uo52krps24agye2z/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/serac-develop-ndjnfsbzzcdi5gp5uo52krps24agye2z/lib64:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/adiak-0.4.0-lren3e7qgebncyc6mcp4urs52tkhdvjn/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/gcc-runtime-10.3.1-yxmwjg76ccmk3rbqtfni7b56ykrldyrb/lib:/usr/tce/packages/mvapich2/mvapich2-2.3.7-gcc-10.3.1/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/axom-0.8.1.1-qshh2cu24cskgqbq7yxm4xw7uzxse4ls/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/conduit-0.8.8-sfqp22ac2h26xxz7v43ela7gr3by5xjj/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/hdf5-1.8.23-nhyxipg27o7hjxhcthkarcvemapz367f/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/zlib-ng-2.1.5-qugqm6afft2zwweu5x5mv6nfxtwljjdc/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/metis-5.1.0-fbbedsqwb2moa7o6vcrwus25yvgrvbwo/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/parmetis-4.0.3-pilepkvz7bmx4tztjs4pvzwzz6sgmqgp/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/mfem-4.6.2-rc0-qtrnq2a7pks4bknkfyiuftktvwdsisqt/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/hypre-2.26.0-4v5i367rmth6kiea4ahqpyujz55pz4v7/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/netcdf-c-4.7.4-otfe5lmy6eqsddhoncjvmgm2lwdjdfpi/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/petsc-3.20.4-2llbwjd25iwoi7shnyo6lxohnqxdbmup/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/netlib-scalapack-2.2.0-m3tuh3di7oy2vyapzdv45ukxo2g677re/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/superlu-dist-8.1.2-4uhpiljdva2keo2wfkrbyikb3ysn42tu/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/slepc-3.20.1-mccjtaiib2hfk6ktr4otg6gebhjungh2/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/raja-2023.06.1-rv75ugta2opfbk7wtzlgw77zjuqgrdi7/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/camp-2023.06.0-iuap4kdanprpfb3bmgefkm5ifat334n3/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/umpire-2023.06.0-ikfxoqkzx6enmo5q4ujgoscyxkcss2gm/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/libunwind-1.6.2-jtxovbckyyaf3da5eewmsfkwnuqsrgko/lib:/usr/tce/packages/clang/clang-14.0.6/lib:/usr/workspace/smithdev/devtools/toss_4_x86_64_ib/latest/python-3.10.12/lib:/usr/tce/packages/python/python-3.9.12/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/tribol-0.1.0.10-pehy2bytgoyibjq6ell622hjktgswquu/lib:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/strumpack-7.2.0-keatppnvs4sl7h3egc452cvkonedei7o/lib64:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/arpack-ng-3.9.0-g5uux4us7d2wo2mhiykvxgpguts4zaiy/lib64:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/sundials-6.7.0-xfn46pedklslbq6pgnxb3rrzo7ickntz/lib64:/usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/caliper-2.10.0-mqa6gv7xqh3hz6v23i4pfifqv4rxq65d/lib64:/collab/usr/global/tools/tce4/packages/gcc/gcc-10.3.1/lib/gcc/x86_64-redhat-linux/10 ../../lib/libgtest_main.a ../../lib/libgtest.a -lpthread ../../lib/libmfem.a /usr/lib64/libopenblas.so -lpthread -lm -ldl /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/strumpack-7.2.0-keatppnvs4sl7h3egc452cvkonedei7o/lib64/libstrumpack.a -lgomp /lib64/libpthread.so /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/sundials-6.7.0-xfn46pedklslbq6pgnxb3rrzo7ickntz/lib64/libsundials_nvecserial.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/sundials-6.7.0-xfn46pedklslbq6pgnxb3rrzo7ickntz/lib64/libsundials_nvecparallel.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/sundials-6.7.0-xfn46pedklslbq6pgnxb3rrzo7ickntz/lib64/libsundials_nvecmpiplusx.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/sundials-6.7.0-xfn46pedklslbq6pgnxb3rrzo7ickntz/lib64/libsundials_cvodes.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/sundials-6.7.0-xfn46pedklslbq6pgnxb3rrzo7ickntz/lib64/libsundials_arkode.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/sundials-6.7.0-xfn46pedklslbq6pgnxb3rrzo7ickntz/lib64/libsundials_kinsol.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/petsc-3.20.4-2llbwjd25iwoi7shnyo6lxohnqxdbmup/lib/libpetsc.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/hypre-2.26.0-4v5i367rmth6kiea4ahqpyujz55pz4v7/lib/libHYPRE.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/strumpack-7.2.0-keatppnvs4sl7h3egc452cvkonedei7o/lib64/libstrumpack.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/netlib-scalapack-2.2.0-m3tuh3di7oy2vyapzdv45ukxo2g677re/lib/libscalapack.so /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/superlu-dist-8.1.2-4uhpiljdva2keo2wfkrbyikb3ysn42tu/lib/libsuperlu_dist.a /usr/lib64/liblapack.so /usr/lib64/libblas.so /usr/lib64/libpthread.so /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/parmetis-4.0.3-pilepkvz7bmx4tztjs4pvzwzz6sgmqgp/lib/libparmetis.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/metis-5.1.0-fbbedsqwb2moa7o6vcrwus25yvgrvbwo/lib/libmetis.a -lgfortran -lm -lgcc_s -lstdc++ -lquadmath /usr/lib64/libdl.so -lslepc -lpetsc /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/netcdf-c-4.7.4-otfe5lmy6eqsddhoncjvmgm2lwdjdfpi/lib/libnetcdf.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/hdf5-1.8.23-nhyxipg27o7hjxhcthkarcvemapz367f/lib/libhdf5_hl.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/hdf5-1.8.23-nhyxipg27o7hjxhcthkarcvemapz367f/lib/libhdf5.a /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/hdf5-1.8.23-nhyxipg27o7hjxhcthkarcvemapz367f/lib/libhdf5.a -lm /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/zlib-ng-2.1.5-qugqm6afft2zwweu5x5mv6nfxtwljjdc/lib/libz.so /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/adiak-0.4.0-lren3e7qgebncyc6mcp4urs52tkhdvjn/lib/libadiak.a -ldl -Wl,-rpath -Wl,/usr/tce/packages/mvapich2/mvapich2-2.3.7-gcc-10.3.1/lib /usr/tce/packages/mvapich2/mvapich2-2.3.7-gcc-10.3.1/lib/libmpifort.so /usr/tce/packages/mvapich2/mvapich2-2.3.7-gcc-10.3.1/lib/libmpicxx.so /usr/tce/packages/mvapich2/mvapich2-2.3.7-gcc-10.3.1/lib/libmpi.so -lgfortran -lquadmath /usr/bin/ld: cannot find -lslepc /usr/bin/ld: cannot find -lpetsc

v-dobrev commented 7 months ago

I'm not 100% sure but I think the flags -lslepc -lpetsc do not originate from MFEM. Can you check the file MFEMTargets.cmake (it should be generated in the MFEM directory) for the properties set with the command

set_target_properties(mfem PROPERTIES

Does the section INTERFACE_LINK_LIBRARIES include -lslepc -lpetsc? From what I see, SLEPc and PETSc are defined there as full paths to the library files, i.e. something like /path/to/slepc/lib/libslepc.a;/path/to/petsc/lib/libpetsc.a. I see such path for PETSc in the command line you posted above:

 /usr/WS2/meemee/serac/serac_libs1/gcc-10.3.1/petsc-3.20.4-2llbwjd25iwoi7shnyo6lxohnqxdbmup/lib/libpetsc.a

However, I do not see such full path for SLEPc. It looks like MFEM_USE_SLEPC maybe OFF.

v-dobrev commented 7 months ago

Regarding the failures when PETSC_EXECUTABLE_RUNS is not set to YES (forcing the PETSc module to skip some checks). The issue is that the checks need the C compiler to be an MPI C wrapper, so typically the checks will run fine if one adds something like -DCMAKE_C_COMPILER=mpicc to the CMake config command line.

chapman39 commented 7 months ago

I'm not 100% sure but I think the flags -lslepc -lpetsc do not originate from MFEM.

Yep... looks like Serac's petsc and slepc cmake modules were conflicting with MFEM's. I'm now only running the MFEM ones in this special codevelop case and it appears to be compiling just fine now.

However, I do not see such full path for SLEPc. It looks like MFEM_USE_SLEPC maybe OFF.

Ah good catch.. looks like I had to set(SLEPC_VERSION_OK "TRUE" CACHE BOOL "") as well to skip its check.

chapman39 commented 7 months ago

Thank you both for the help. Since this issue was due to some build mistakes I was making rather than MFEM itself, I'll be closing this issue.