search

trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.18k stars 559 forks source link

missing fftw routine during compiler #10322

Closed VictorEijkhout closed 1 year ago

VictorEijkhout commented 2 years ago 
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: /opt/apps/intel19/impi19_0/fftw3/3.3.8/lib/libfftw3_mpi.so.3: undefined reference to `fftw_ialignment_of'
/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/kokkos/core/src/Kokkos_Parallel.hpp(163): remark #15009: _ZN6Kokkos12parallel_forINS_10TeamPolicyIJNS_6SerialEEEEN12KokkosSparse4Impl16PointGaussSeidelIN13KokkosKernels12Experimental19KokkosKernelsHandleIKiSA_KdS2_NS_9HostSpaceESC_EENS_4ViewIPSA_JNS_10LayoutLeftENS_6DeviceIS2_SC_EENS_12MemoryTraitsILj1EEEEEESL_NSE_IPSB_JSG_SI_SK_EEEE19fill_matrix_numericEEEvRKT_RKT0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNSt9enable_ifIXsr6Kokkos19is_execution_policyISQ_EE5valueEvE4typeE has been targeted for automatic cpu dispatch
make[2]: *** [packages/intrepid/test/Discretization/Basis/HCURL_HEX_In_FEM/Intrepid_test_Discretization_Basis_HCURL_HEX_In_FEM_Test_02.exe] Error 1

Just let me know what log file to send you. (Why is there a missing fftw routine while I'm not specifying anything fftw in the configuration?)

jhux2 commented 2 years ago

@trilinos/intrepid

mperego commented 2 years ago

I don't understand. Intrepid does not depend on Kokkos at all. I don't see how this could be an issue with Intrepid.

mperego commented 2 years ago

@VictorEijkhout do you need Intrepid package or can you simply disable it?

cgcgcg commented 2 years ago

Do we have anything using fftw in Trilinos? I don't think so. Maybe this is coming from somewhere else?

VictorEijkhout commented 2 years ago

If I had to guess: you're linking petsc, petsc uses fftw, but you're not properly reading the petsc link variables.

VictorEijkhout commented 2 years ago

Just curious: how are you determining the petsc library to link to? I'm starting to discover the beauty of .pc files.

mperego commented 2 years ago

I'm removing the Intrepid label from this issue, as I don't think it's related to it. It's not clear it's a bug either.

You don't need to link petsc in order to build Trilinos. Maybe the @trilinos/framework can help. I'd suggest that you paste your configuration script and tell us what you are trying to do (what package do you need, what problem you are trying to solve).

VictorEijkhout commented 2 years ago

You're right that petsc is not needed. I use the hdf5 and mumps from petsc. Does mumps use fftw?

Which of the log files contains the configuration?

VictorEijkhout commented 2 years ago

It doesn't look like mumps uses fftw. Can your build print out the full link line that leads to this problem?

cgcgcg commented 2 years ago

You could try make VERBOSE=1.

VictorEijkhout commented 2 years ago

Ok. 1. I was mistakenly linking petsc with the scotch/mumps libraries. Removed.

  1. cd /tmp/trilinos-build/packages/intrepid/test/Discretization/Basis/HDIV_HEX_In_FEM && /opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicxx  -I/tmp/trilinos-build -I/\tmp/trilinos-build/packages/intrepid/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages\/intrepid/src/Shared -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src/Discretization/Integration -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/p\ackages/intrepid/src/Discretization/FunctionSpaceTools -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src/Discretization/Basis -I/admin/build/admin/rpms/stamp\ede2/BUILD/trilinos-13.0.1/packages/intrepid/src/Discretization/TensorProductSpaceTools -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src/Cell -I/tmp/trilino\s-build/packages/shards/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/shards/src -I/tmp/trilinos-build/packages/sacado/src -I/admin/build/admin/rpms/stampede2/BUI\LD/trilinos-13.0.1/packages/sacado/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/sacado/src/new_design -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/p\ackages/sacado/src/template -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/sacado/src/parameter -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/saca\do/src/mpl -I/tmp/trilinos-build/packages/teuchos/kokkoscomm/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/teuchos/kokkoscomm/src -I/tmp/trilinos-build/packages/t\euchos/kokkoscompat/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/teuchos/kokkoscompat/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/teuc\hos/parameterlist/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/teuchos/parser/src -I/tmp/trilinos-build/packages/teuchos/core/src -I/admin/build/admin/rpms/stamp\ede2/BUILD/trilinos-13.0.1/packages/teuchos/core/src -I/tmp/trilinos-build/packages/kokkos/core/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/kokkos/core/src -I/t\mp/trilinos-build/packages/kokkos -I/opt/apps/intel19/python3_7/boost/1.72/include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/teuchos/comm/src -I/admin/build/admin\/rpms/stampede2/BUILD/trilinos-13.0.1/packages/teuchos/numerics/src -I/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13\.0.1/packages/teuchos/remainder/src -I/tmp/trilinos-build/packages/teuchos/remainder/src -I/tmp/trilinos-build/packages/pamgen/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.\1/packages/pamgen/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/pamgen/mesh_spec_lt -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/pamgen/extr\a_functions -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/pamgen/rtcompiler -I/tmp/trilinos-build/packages/amesos/src -I/admin/build/admin/rpms/stampede2/BUILD/trilin\os-13.0.1/packages/amesos/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/AMD/Include -I/admin/build/admin/rpms/stampede2/B\UILD/trilinos-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/BTF/Include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/\src/CAMD/Include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/CCOLAMD/Include -I/admin/build/admin/rpms/stampede2/BUILD/tril\
inos-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/COLAMD/Include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/KL\
U/Include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/UFconfig -I/tmp/trilinos-build/packages/epetraext/src -I/admin/build/\
admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetraext/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetraext/src/transform -I/admin/build/admin/rpms/stam\
pede2/BUILD/trilinos-13.0.1/packages/epetraext/src/inout -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetraext/src/coloring -I/admin/build/admin/rpms/stampede2/BUIL\
D/trilinos-13.0.1/packages/epetraext/src/model_evaluator -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetraext/src/block -I/admin/build/admin/rpms/stampede2/BUILD/t\
rilinos-13.0.1/packages/epetraext/src/restrict -I/opt/apps/intel19/python3/3.7.0/include/python3.7m -I/tmp/trilinos-build/packages/triutils/src -I/admin/build/admin/rpms/stampede2/BUILD/t\
rilinos-13.0.1/packages/triutils/src -I/tmp/trilinos-build/packages/epetra/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetra/src -I/home1/apps/intel19/impi19_0\
/petsc/3.16/skylake/include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/commonTools/gtest -g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -mkl -DMPICH_SKIP_MPICXX   -O3 -DNDEBU\
G -std=c++11 -MD -MT packages/intrepid/test/Discretization/Basis/HDIV_HEX_In_FEM/CMakeFiles/Intrepid_test_Discretization_Basis_HDIV_HEX_In_FEM_Test_02.dir/test_02.cpp.o -MF CMakeFiles/Int\
repid_test_Discretization_Basis_HDIV_HEX_In_FEM_Test_02.dir/test_02.cpp.o.d -o CMakeFiles/Intrepid_test_Discretization_Basis_HDIV_HEX_In_FEM_Test_02.dir/test_02.cpp.o -c /admin/build/admi\
n/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/test/Discretization/Basis/HDIV_HEX_In_FEM/test_02.cpp
/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src/Shared/Intrepid_FieldContainerDef.hpp(124): remark #15009: _ZN8Intrepid14FieldContainerIdLi0EEC1Eiii has been\
targeted for automatic cpu dispatch
/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/kokkos-kernels/src/sparse/impl/KokkosSparse_spmv_struct_impl.hpp(1220): remark #15009: _ZNK12KokkosSparse4Impl33SPMV_MV_St\
ruct_LayoutLeft_FunctorINS_9CrsMatrixIKdKiN6Kokkos6DeviceINS5_6SerialENS5_9HostSpaceEEENS5_12MemoryTraitsILj1EEES4_EENS5_4ViewIPPS3_JNS5_10LayoutLeftES9_NSA_ILj3EEEEEENSD_IPPdJSG_S9_SB_EE\
ELin1ELi2ELb0EEclERKNS5_4Impl20HostThreadTeamMemberIS7_EE has been targeted for automatic cpu dispatch
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `blacs_gridinfo_'
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `_SCOTCHstringSubst'
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `_SCOTCHintPerm'
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `esmumpsv'

Which log file can I send you for my configuration? Oh never mind, I'll just include it because I suspect that cmake doesn't actually record how it's being called. Correct me if I'm wrong.

+ cmake -D BUILD_SHARED_LIBS:BOOL=ON -D Trilinos_VERBOSE_CONFIGURE=OFF -D CMAKE_VERBOSE_MAKEFILE=ON -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF -D Trilinos_ASSERT_MISSING_PACKAGES=OFF -D Tri\
linos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF -D Trilinos_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_EXAMPLES:BOOL=ON -D Trilinos_ENABLE_Export_Makefiles:BOOL=ON -D Trilinos_ENABLE_Fortran:BO\
OL=ON -D CMAKE_INSTALL_PREFIX:PATH=/home1/apps/intel19/impi19_0/trilinos/13.0.1 -D CMAKE_BUILD_TYPE:STRING=RELEASE -D 'CMAKE_C_FLAGS:STRING=-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -m\
kl' -D 'CMAKE_CXX_FLAGS:STRING=-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -mkl -DMPICH_SKIP_MPICXX' -D BLAS_INCLUDE_DIRS:PATH=/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/inc\
lude -D BLAS_LIBRARY_DIRS:PATH=/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin -D 'BLAS_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_\
scalapack_lp64;libmkl_blacs_intelmpi_lp64' -D LAPACK_INCLUDE_DIRS:PATH=/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/include -D LAPACK_LIBRARY_DIRS:PATH=/opt/intel/compilers_and\
_libraries_2020.1.217/linux/mkl/lib/intel64_lin -D 'LAPACK_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64' -D TPL_ENA\
BLE_MPI:BOOL=ON -D MPI_BASE_DIR=/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64 -D MPI_INCLUDE_DIRS=/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/include\
 -D MPI_EXEC:FILEPATH=/opt/apps/xalt/0.6/bin/ibrun -D TPL_MPI_BASE_DIR=/opt/intel/compilers_and_libraries_2020.4.304/linux/mpiintel64 -D TPL_MPI_INCLUDE_DIRS=/opt/intel/compilers_and_libr\
aries_2020.4.304/linux/mpi/intel64/include -D TPL_ENABLE_GLM=OFF -D TPL_ENABLE_Matio=OFF -D TPL_ENABLE_MUMPS:BOOL=ON -D MUMPS_INCLUDE_DIRS=/home1/apps/intel19/impi19_0/petsc/3.16/skylake/\
include -D 'MUMPS_LIBRARY_DIRS=/home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib;/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl;/opt/intel/compilers_and_libraries_2020.1.217/li\
nux/mkl/../compiler/lib/intel64_lin' -D 'MUMPS_LIBRARY_NAMES:STRING=ifcore;dmumps;mumps_common;ptscotch;ptscotcherr;ptscotcherrexit;ptscotchparmetis;pord;mkl_scalapack_lp64;mkl_intel_lp64\
;mkl_intel_thread;mkl_core;iomp5;mkl_blacs_intelmpi_lp64;pthread;ifcore' -D VLE_TPL_MUMPS_LIBRARIES:STRING= dmumps mumps_common ptscotch ptscotcherr ptscotcherrexit ptscotchparmetis pord \
mkl_scalapack_lp64 mkl_intel_lp64 mkl_intel_thread mkl_core iomp5 mkl_blacs_intelmpi_lp64 pthread ifcore -D 'TPL_ParMETIS_LIBRARIES=/libptscotchparmetis.a;/libparmetis.so' -D TPL_ParMETIS\
_INCLUDE_DIRS= -D TPL_ENABLE_Boost:BOOL=ON -D Boost_INCLUDE_DIRS:PATH=/opt/apps/intel19/python3_7/boost/1.72/include -D Boost_LIBRARY_DIRS:PATH=/opt/apps/intel19/python3_7/boost/1.72/lib \
-D TPL_ENABLE_BoostLib:BOOL=ON -D BoostLib_INCLUDE_DIRS:PATH=/opt/apps/intel19/python3_7/boost/1.72/include -D BoostLib_LIBRARY_DIRS:PATH=/opt/apps/intel19/python3_7/boost/1.72/lib -D TPL\
_ENABLE_HDF5:BOOL=ON -D HDF5_INCLUDE_DIRS:PATH= -D HDF5_LIBRARY_DIRS:PATH= -D TPL_ENABLE_Netcdf:BOOL=ON -D Netcdf_INCLUDE_DIRS:PATH=/opt/apps/intel19/impi19_0/parallel-netcdf/4.6.2/x86_64\
/include -D Netcdf_LIBRARY_DIRS:PATH=/opt/apps/intel19/impi19_0/parallel-netcdf/4.6.2/x86_64/lib -D VLE_SUPERLU_IS_FIXED_AFTER_12.12:BOOL=ON -D TPL_ENABLE_SuperLU:BOOL=OFF -D SuperLU_INCL\
UDE_DIRS:PATH=/home/foo/include -D SuperLU_LIBRARY_DIRS:PATH=/home/foo -D SuperLU_LIBRARY_NAMES:STRING=superlu -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON -D Trilinos_ENABLE_Amesos:B\
OOL=ON -D Trilinos_ENABLE_Amesos2:BOOL=ON -D Amesos2_ENABLE_KLU2:BOOL=ON -D Amesos2_ENABLE_Basker:BOOL=ON -D Trilinos_ENABLE_Anasazi:BOOL=ON -D Trilinos_ENABLE_AztecOO:Bool=ON -D Trilinos\
_ENABLE_Belos:BOOL=ON -D Trilinos_ENABLE_Epetra:Bool=ON -D Trilinos_ENABLE_EpetraExt:Bool=ON -D Epetra_ENABLE_TESTS:BOOL=ON -D Trilinos_ENABLE_FEI:Bool=ON -D Trilinos_ENABLE_Ifpack:Bool=O\
N -D Trilinos_ENABLE_Ifpack2:BOOL=ON -D Trilinos_ENABLE_Intrepid:BOOL=ON -D Trilinos_ENABLE_Intrepid2:BOOL=ON -D Intrepid_ENABLE_TESTS:BOOL=ON -D Trilinos_ENABLE_Issoropia:BOOL=ON -D Tril\
inos_ENABLE_ML:BOOL=ON -D ML_TAKES_SUPERLU_LESS_THAN_5=TRUE -D ML_ENABLE_SuperLU:BOOL=OFF -D Trilinos_ENABLE_MOOCHO:BOOL=ON -D Trilinos_ENABLE_MueLu:BOOL=ON -D MueLu_ENABLE_Tutorial:BOOL=\
OFF -D MueLu_ENABLE_EXAMPLES:BOOL=OFF -D Trilinos_ENABLE_NOX=ON -D NOX_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_Pamgen:Bool=ON -D Pamgen_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_Panzer:Bo\
ol=ON -D Trilinos_ENABLE_Phalanx:BOOL=ON -D Phalanx_EXPLICIT_TEMPLATE_INSTANTIATION=ON -D Phalanx_ENABLE_EXAMPLES=OFF -D Trilinos_ENABLE_Piro:BOOL=ON -D Trilinos_ENABLE_Rythmos:BOOL=ON -D\
 Trilinos_ENABLE_Sacado:Bool=ON -D Trilinos_ENABLE_SEACAS:BOOL=ON -D Trilinos_ENABLE_SEACASIoss:BOOL=ON -D Trilinos_ENABLE_SEACASBlot:BOOL=ON -D Trilinos_ENABLE_SEACASExodus:BOOL=ON -D Tr\
ilinos_ENABLE_SECONDARY_STABLE_CODE:BOOL=ON -D Trilinos_ENABLE_Shards:BOOL=ON -D Trilinos_ENABLE_ShyLU:BOOL=ON -D Trilinos_ENABLE_Stokhos:BOOL=ON -D Trilinos_ENABLE_Stratimikos:BOOL=ON -D\
 Trilinos_ENABLE_Teko:BOOL=ON -D Trilinos_ENABLE_Teuchos:BOOL=ON -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON -D Trilinos_ENABLE_Thyra:BOOL=ON -D Trilinos_ENABLE_Tpetra:BOOL=ON -D Trilinos_ENA\
BLE_TriKota:BOOL=ON -D Trilinos_ENABLE_Zoltan:BOOL=ON -D Trilinos_ENABLE_Zoltan2:BOOL=ON -D Trilinos_ENABLE_Kokkos:BOOL=ON -D Trilinos_ENABLE_KokkosCore:BOOL=ON -D Phalanx_KOKKOS_DEVICE_T\
YPE:STRING=SERIAL -D Phalanx_INDEX_SIZE_TYPE:STRING=INT -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF -D Kokkos_ENABLE_Serial:BOOL=ON -D Kokkos_ENABLE_OpenMP:BOOL=OFF -D Kokkos_ENABLE_Pthr\
ead:BOOL=OFF -D Trilinos_ENABLE_STK:BOOL=ON -D Trilinos_ENABLE_STKIO:BOOL=ON -D Trilinos_ENABLE_STKMesh:BOOL=ON -D Trilinos_ENABLE_PyTrilinos:Bool=ON -D CMAKE_PYTHON_INCLUDE_DIR:PATH=/opt\
/apps/intel19/python3/3.7.0/include -D CMAKE_PYTHON_LIBRARIES:STRING=/opt/apps/intel19/python3/3.7.0/lib -D CMAKE_PYTHON_EXECUTABLE:FILEPATH=/opt/apps/intel19/python3/3.7.0/bin/python3 -D\
 PYTHON_EXECUTABLE:PATH=/opt/apps/intel19/python3/3.7.0/bin/python3 -D PyTrilinos_DOCSTRINGS:BOOL=OFF -D PyTrilinos_ENABLE_Tpetra:BOOL=OFF -D PyTrilinos_ENABLE_Teuchos:BOOL=OFF -D SWIG_EX\
ECUTABLE:FILEPATH=/home1/apps/intel19/swig/4.0.2/bin/swig /admin/build/admin/rpms/stampede2/BUILD//trilinos-13.0.1
cgcgcg commented 2 years ago

@trilinos/amesos Looks like there is a problem linking with MUMPS, BLACS and SCOTCH.

VictorEijkhout commented 2 years ago

I'm now at the point where scalapack / blacs stuff is missing:

/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `blacs_gridinfo_'
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `pdpotrf_'
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `pdgetrs_'

even though I think the right MKL libraries are provided:

+ cmake -D BUILD_SHARED_LIBS:BOOL=ON -D Trilinos_VERBOSE_CONFIGURE=OFF -D CMAKE_VERBOSE_MAKEFILE=ON -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF -D Trilinos_ASSERT_MISSING_PACKAGES=OFF -D Tri\
linos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF -D Trilinos_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_EXAMPLES:BOOL=ON -D Trilinos_ENABLE_Export_Makefiles:BOOL=ON -D Trilinos_ENABLE_Fortran:BO\
OL=ON -D CMAKE_INSTALL_PREFIX:PATH=/home1/apps/intel19/impi19_0/trilinos/13.0.1 -D CMAKE_BUILD_TYPE:STRING=RELEASE -D 'CMAKE_C_FLAGS:STRING=-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -m\
kl' -D 'CMAKE_CXX_FLAGS:STRING=-g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -mkl -DMPICH_SKIP_MPICXX' -D BLAS_INCLUDE_DIRS:PATH=/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/inc\
lude -D BLAS_LIBRARY_DIRS:PATH=/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin -D 'BLAS_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_\
scalapack_lp64;libmkl_blacs_intelmpi_lp64' -D LAPACK_INCLUDE_DIRS:PATH=/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/include -D LAPACK_LIBRARY_DIRS:PATH=/opt/intel/compilers_and\
_libraries_2020.1.217/linux/mkl/lib/intel64_lin -D 'LAPACK_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64' -D TPL_ENA\
BLE_MPI:BOOL=ON -D MPI_BASE_DIR=/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64 -D MPI_INCLUDE_DIRS=/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/include\
 -D MPI_EXEC:FILEPATH=/opt/apps/xalt/0.6/bin/ibrun -D TPL_MPI_BASE_DIR=/opt/intel/compilers_and_libraries_2020.4.304/linux/mpiintel64 -D TPL_MPI_INCLUDE_DIRS=/opt/intel/compilers_and_libr\
aries_2020.4.304/linux/mpi/intel64/include -D TPL_ENABLE_GLM=OFF -D TPL_ENABLE_Matio=OFF -D TPL_ENABLE_MUMPS:BOOL=ON -D MUMPS_INCLUDE_DIRS=/home1/apps/intel19/impi19_0/petsc/3.16/skylake/\
include -D 'MUMPS_LIBRARY_DIRS=/home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib;/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl;/opt/intel/compilers_and_libraries_2020.1.217/li\
nux/mkl/../compiler/lib/intel64_lin' -D 'MUMPS_LIBRARY_NAMES:STRING=ifcore;dmumps;esmumps;mumps_common;ptscotch;ptscotcherr;ptscotcherrexit;ptscotchparmetis;scotch;pord;mkl_scalapack_lp64\
;mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5;mkl_blacs_intelmpi_lp64;pthread;ifcore' -D VLE_TPL_MUMPS_LIBRARIES:STRING= dmumps esmumps mumps_common ptscotch ptscotcherr ptscotcherrexit\
 ptscotchparmetis scotch pord mkl_scalapack_lp64 mkl_intel_lp64 mkl_intel_thread mkl_core iomp5 mkl_blacs_intelmpi_lp64 pthread ifcore -D 'TPL_ParMETIS_LIBRARIES=/libptscotchparmetis.a;/l\
ibparmetis.so' -D TPL_ParMETIS_INCLUDE_DIRS= -D TPL_ENABLE_Boost:BOOL=ON -D Boost_INCLUDE_DIRS:PATH=/opt/apps/intel19/python3_7/boost/1.72/include -D Boost_LIBRARY_DIRS:PATH=/opt/apps/int\
el19/python3_7/boost/1.72/lib -D TPL_ENABLE_BoostLib:BOOL=ON -D BoostLib_INCLUDE_DIRS:PATH=/opt/apps/intel19/python3_7/boost/1.72/include -D BoostLib_LIBRARY_DIRS:PATH=/opt/apps/intel19/p\
ython3_7/boost/1.72/lib -D TPL_ENABLE_HDF5:BOOL=ON -D HDF5_INCLUDE_DIRS:PATH= -D HDF5_LIBRARY_DIRS:PATH= -D TPL_ENABLE_Netcdf:BOOL=ON -D Netcdf_INCLUDE_DIRS:PATH=/opt/apps/intel19/impi19_\
0/parallel-netcdf/4.6.2/x86_64/include -D Netcdf_LIBRARY_DIRS:PATH=/opt/apps/intel19/impi19_0/parallel-netcdf/4.6.2/x86_64/lib -D VLE_SUPERLU_IS_FIXED_AFTER_12.12:BOOL=ON -D TPL_ENABLE_Su\
perLU:BOOL=OFF -D SuperLU_INCLUDE_DIRS:PATH=/home/foo/include -D SuperLU_LIBRARY_DIRS:PATH=/home/foo -D SuperLU_LIBRARY_NAMES:STRING=superlu -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=\
ON -D Trilinos_ENABLE_Amesos:BOOL=ON -D Trilinos_ENABLE_Amesos2:BOOL=ON -D Amesos2_ENABLE_KLU2:BOOL=ON -D Amesos2_ENABLE_Basker:BOOL=ON -D Trilinos_ENABLE_Anasazi:BOOL=ON -D Trilinos_ENAB\
LE_AztecOO:Bool=ON -D Trilinos_ENABLE_Belos:BOOL=ON -D Trilinos_ENABLE_Epetra:Bool=ON -D Trilinos_ENABLE_EpetraExt:Bool=ON -D Epetra_ENABLE_TESTS:BOOL=ON -D Trilinos_ENABLE_FEI:Bool=ON -D\
 Trilinos_ENABLE_Ifpack:Bool=ON -D Trilinos_ENABLE_Ifpack2:BOOL=ON -D Trilinos_ENABLE_Intrepid:BOOL=ON -D Trilinos_ENABLE_Intrepid2:BOOL=ON -D Intrepid_ENABLE_TESTS:BOOL=ON -D Trilinos_EN\
ABLE_Issoropia:BOOL=ON -D Trilinos_ENABLE_ML:BOOL=ON -D ML_TAKES_SUPERLU_LESS_THAN_5=TRUE -D ML_ENABLE_SuperLU:BOOL=OFF -D Trilinos_ENABLE_MOOCHO:BOOL=ON -D Trilinos_ENABLE_MueLu:BOOL=ON \
-D MueLu_ENABLE_Tutorial:BOOL=OFF -D MueLu_ENABLE_EXAMPLES:BOOL=OFF -D Trilinos_ENABLE_NOX=ON -D NOX_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_Pamgen:Bool=ON -D Pamgen_ENABLE_TESTS:BOOL=OF\
F -D Trilinos_ENABLE_Panzer:Bool=ON -D Trilinos_ENABLE_Phalanx:BOOL=ON -D Phalanx_EXPLICIT_TEMPLATE_INSTANTIATION=ON -D Phalanx_ENABLE_EXAMPLES=OFF -D Trilinos_ENABLE_Piro:BOOL=ON -D Tril\
inos_ENABLE_Rythmos:BOOL=ON -D Trilinos_ENABLE_Sacado:Bool=ON -D Trilinos_ENABLE_SEACAS:BOOL=ON -D Trilinos_ENABLE_SEACASIoss:BOOL=ON -D Trilinos_ENABLE_SEACASBlot:BOOL=ON -D Trilinos_ENA\
BLE_SEACASExodus:BOOL=ON -D Trilinos_ENABLE_SECONDARY_STABLE_CODE:BOOL=ON -D Trilinos_ENABLE_Shards:BOOL=ON -D Trilinos_ENABLE_ShyLU:BOOL=ON -D Trilinos_ENABLE_Stokhos:BOOL=ON -D Trilinos\
_ENABLE_Stratimikos:BOOL=ON -D Trilinos_ENABLE_Teko:BOOL=ON -D Trilinos_ENABLE_Teuchos:BOOL=ON -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON -D Trilinos_ENABLE_Thyra:BOOL=ON -D Trilinos_ENABLE_\
Tpetra:BOOL=ON -D Trilinos_ENABLE_TriKota:BOOL=ON -D Trilinos_ENABLE_Zoltan:BOOL=ON -D Trilinos_ENABLE_Zoltan2:BOOL=ON -D Trilinos_ENABLE_Kokkos:BOOL=ON -D Trilinos_ENABLE_KokkosCore:BOOL\
=ON -D Phalanx_KOKKOS_DEVICE_TYPE:STRING=SERIAL -D Phalanx_INDEX_SIZE_TYPE:STRING=INT -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF -D Kokkos_ENABLE_Serial:BOOL=ON -D Kokkos_ENABLE_OpenMP:\
BOOL=OFF -D Kokkos_ENABLE_Pthread:BOOL=OFF -D Trilinos_ENABLE_STK:BOOL=ON -D Trilinos_ENABLE_STKIO:BOOL=ON -D Trilinos_ENABLE_STKMesh:BOOL=ON -D Trilinos_ENABLE_PyTrilinos:Bool=ON -D CMAK\
E_PYTHON_INCLUDE_DIR:PATH=/opt/apps/intel19/python3/3.7.0/include -D CMAKE_PYTHON_LIBRARIES:STRING=/opt/apps/intel19/python3/3.7.0/lib -D CMAKE_PYTHON_EXECUTABLE:FILEPATH=/opt/apps/intel1\
9/python3/3.7.0/bin/python3 -D PYTHON_EXECUTABLE:PATH=/opt/apps/intel19/python3/3.7.0/bin/python3 -D PyTrilinos_DOCSTRINGS:BOOL=OFF -D PyTrilinos_ENABLE_Tpetra:BOOL=OFF -D PyTrilinos_ENAB\
LE_Teuchos:BOOL=OFF -D SWIG_EXECUTABLE:FILEPATH=/home1/apps/intel19/swig/4.0.2/bin/swig /admin/build/admin/rpms/stampede2/BUILD//trilinos-13.0.1
jhux2 commented 2 years ago

@VictorEijkhout Please check the actual link line. It could be that the libraries are incorrectly order.

VictorEijkhout commented 2 years ago

Here you go:

 65%] Building CXX object packages/intrepid/test/Discretization/Basis/HCURL_QUAD_I1_FEM/CMakeFiles/Intrepid_test_Discretization_Basis_HCURL_QUAD_I1_FEM_Test_01.di\
r/test_01.cpp.o
cd /tmp/trilinos-build/packages/intrepid/test/Discretization/Basis/HCURL_QUAD_I1_FEM && /opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicxx \
 -I/tmp/trilinos-build -I/tmp/trilinos-build/packages/intrepid/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src -I/admin/build/a\
dmin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src/Shared -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src/Discretizati\
on/Integration -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src/Discretization/FunctionSpaceTools -I/admin/build/admin/rpms/stampede\
2/BUILD/trilinos-13.0.1/packages/intrepid/src/Discretization/Basis -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src/Discretization/T\
ensorProductSpaceTools -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/src/Cell -I/tmp/trilinos-build/packages/shards/src -I/admin/buil\
d/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/shards/src -I/tmp/trilinos-build/packages/sacado/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.\
1/packages/sacado/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/sacado/src/new_design -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-1\
3.0.1/packages/sacado/src/template -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/sacado/src/parameter -I/admin/build/admin/rpms/stampede2/BUIL\
D/trilinos-13.0.1/packages/sacado/src/mpl -I/tmp/trilinos-build/packages/teuchos/kokkoscomm/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/\
teuchos/kokkoscomm/src -I/tmp/trilinos-build/packages/teuchos/kokkoscompat/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/teuchos/kokkoscom\
pat/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/teuchos/parameterlist/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/pack\
ages/teuchos/parser/src -I/tmp/trilinos-build/packages/teuchos/core/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/teuchos/core/src -I/tmp/\
trilinos-build/packages/kokkos/core/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/kokkos/core/src -I/tmp/trilinos-build/packages/kokkos -I\
/opt/apps/intel19/python3_7/boost/1.72/include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/teuchos/comm/src -I/admin/build/admin/rpms/stampe\
de2/BUILD/trilinos-13.0.1/packages/teuchos/numerics/src -I/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/include -I/admin/build/admin/rpms/stampede2/BUILD\
/trilinos-13.0.1/packages/teuchos/remainder/src -I/tmp/trilinos-build/packages/teuchos/remainder/src -I/tmp/trilinos-build/packages/pamgen/src -I/admin/build/admin\
/rpms/stampede2/BUILD/trilinos-13.0.1/packages/pamgen/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/pamgen/mesh_spec_lt -I/admin/build/adm\
in/rpms/stampede2/BUILD/trilinos-13.0.1/packages/pamgen/extra_functions -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/pamgen/rtcompiler -I/tmp\
/trilinos-build/packages/amesos/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/amesos/src -I/admin/build/admin/rpms/stampede2/BUILD/trilino\
s-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/AMD/Include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/common/auxiliarySoftware/\
SuiteSparse/src/BTF/Include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/CAMD/Include -I/admin/build\
/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/CCOLAMD/Include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13\
.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/COLAMD/Include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/common/auxiliarySoftware/S\
uiteSparse/src/KLU/Include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/common/auxiliarySoftware/SuiteSparse/src/UFconfig -I/tmp/trilinos-bui\
ld/packages/epetraext/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetraext/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.\
1/packages/epetraext/src/transform -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetraext/src/inout -I/admin/build/admin/rpms/stampede2/BUILD\
/trilinos-13.0.1/packages/epetraext/src/coloring -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetraext/src/model_evaluator -I/admin/build/ad\
min/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetraext/src/block -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetraext/src/restrict -I/\
opt/apps/intel19/python3/3.7.0/include/python3.7m -I/tmp/trilinos-build/packages/triutils/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/tr\
iutils/src -I/tmp/trilinos-build/packages/epetra/src -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/epetra/src -I/home1/apps/intel19/impi19_0/p\
etsc/3.16/skylake/include -I/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/commonTools/gtest -g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -mkl -DMPICH_SKI\
P_MPICXX   -O3 -DNDEBUG -std=c++11 -MD -MT packages/intrepid/test/Discretization/Basis/HCURL_QUAD_I1_FEM/CMakeFiles/Intrepid_test_Discretization_Basis_HCURL_QUAD_I\
1_FEM_Test_01.dir/test_01.cpp.o -MF CMakeFiles/Intrepid_test_Discretization_Basis_HCURL_QUAD_I1_FEM_Test_01.dir/test_01.cpp.o.d -o CMakeFiles/Intrepid_test_Discret\
ization_Basis_HCURL_QUAD_I1_FEM_Test_01.dir/test_01.cpp.o -c /admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/intrepid/test/Discretization/Basis/HC\
URL_QUAD_I1_FEM/test_01.cpp

As you can see, there are no blacs libraries listed even though I did specify them in the configuration.

jhux2 commented 2 years ago

I think you want to look at the link line, no?

VictorEijkhout commented 2 years ago

Ah. I grabbed the last command before the error. However with a parallel make that is not necessarily what I want.

You know what, I'll send you the whole log output. You're probably better at finding the offending command than I am. amesos.zip

(Alternatively I can do a single threaded make and I'll get back to you in about 3 days.)

VictorEijkhout commented 2 years ago

Somehow the serial build crahsed earlier than the parallel.

[ 19%] Linking CXX executable Amesos_example_AmesosFactory.exe
cd /tmp/trilinos-build/packages/amesos/example && /opt/apps/cmake/3.20.3/bin/cmake -E cmake_link_script CMakeFiles/Amesos_example_AmesosFactory.dir/link.txt --verb\
ose=1
/opt/intel/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicxx -g -xCORE-AVX2 -axMIC-AVX512,CORE-AVX512 -O2 -mkl -DMPICH_SKIP_MPICXX   -O3 -DNDEBUG CMa\
keFiles/Amesos_example_AmesosFactory.dir/example_AmesosFactory.cpp.o -o Amesos_example_AmesosFactory.exe  -Wl,-rpath,/tmp/trilinos-build/packages/amesos/src:/tmp/t\
rilinos-build/packages/galeri/src-xpetra:/tmp/trilinos-build/packages/galeri/src-epetra:/tmp/trilinos-build/packages/common/auxiliarySoftware/SuiteSparse/src:/tmp/\
trilinos-build/packages/xpetra/sup:/tmp/trilinos-build/packages/xpetra/src:/tmp/trilinos-build/packages/thyra/adapters/tpetra/src:/tmp/trilinos-build/packages/thyr\
a/adapters/epetraext/src:/tmp/trilinos-build/packages/epetraext/src:/tmp/trilinos-build/packages/triutils/src:/home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib:/\
tmp/trilinos-build/packages/thyra/adapters/epetra/src:/tmp/trilinos-build/packages/thyra/core/src:/tmp/trilinos-build/packages/rtop/src:/tmp/trilinos-build/package\
s/tpetra/core/ext:/tmp/trilinos-build/packages/tpetra/core/inout:/tmp/trilinos-build/packages/tpetra/core/src:/tmp/trilinos-build/packages/epetra/src:/tmp/trilinos\
-build/packages/tpetra/tsqr/src:/tmp/trilinos-build/packages/kokkos-kernels/src:/tmp/trilinos-build/packages/kokkos/algorithms/src:/tmp/trilinos-build/packages/kok\
kos/containers/src:/tmp/trilinos-build/packages/tpetra/classic/LinAlg:/tmp/trilinos-build/packages/tpetra/classic/NodeAPI:/tmp/trilinos-build/packages/tpetra/class\
ic/src:/tmp/trilinos-build/packages/teuchos/kokkoscomm/src:/tmp/trilinos-build/packages/teuchos/kokkoscompat/src:/tmp/trilinos-build/packages/teuchos/remainder/src\
:/tmp/trilinos-build/packages/teuchos/numerics/src:/tmp/trilinos-build/packages/teuchos/comm/src:/tmp/trilinos-build/packages/teuchos/parameterlist/src:/tmp/trilin\
os-build/packages/teuchos/parser/src:/tmp/trilinos-build/packages/teuchos/core/src:/tmp/trilinos-build/packages/kokkos/core/src ../src/libamesos.so.13.0 ../../gale\
ri/src-xpetra/libgaleri-xpetra.so.13.0 ../../galeri/src-epetra/libgaleri-epetra.so.13.0 ../../common/auxiliarySoftware/SuiteSparse/src/libtrilinosss.so.13.0 -lm -l\
ifcore /home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib/libdmumps.a /home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib/libesmumps.a /home1/apps/intel19/impi19_\
0/petsc/3.16/skylake/lib/libmumps_common.a /home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib/libptscotch.a /home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib/li\
bptscotcherr.a /home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib/libptscotcherrexit.a /home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib/libptscotchparmetis.a /\
home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib/libscotch.a /home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib/libpord.a ../../xpetra/sup/libxpetra-sup.so.13.0\
 ../../xpetra/src/libxpetra.so.13.0 ../../thyra/adapters/tpetra/src/libthyratpetra.so.13.0 ../../thyra/adapters/epetraext/src/libthyraepetraext.so.13.0 ../../epetr\
aext/src/libepetraext.so.13.0 ../../triutils/src/libtriutils.so.13.0 /home1/apps/intel19/impi19_0/petsc/3.16/skylake/lib/libhdf5.so /usr/lib64/libm.so ../../thyra/\
adapters/epetra/src/libthyraepetra.so.13.0 ../../thyra/core/src/libthyracore.so.13.0 ../../rtop/src/librtop.so.13.0 ../../tpetra/core/ext/libtpetraext.so.13.0 ../.\
./tpetra/core/inout/libtpetrainout.so.13.0 ../../tpetra/core/src/libtpetra.so.13.0 ../../epetra/src/libepetra.so.13.0 ../../tpetra/tsqr/src/libkokkostsqr.so.13.0 .\
./../kokkos-kernels/src/libkokkoskernels.so.13.0 ../../kokkos/algorithms/src/libkokkosalgorithms.so.13.0 ../../kokkos/containers/src/libkokkoscontainers.so.3.1.1 .\
./../tpetra/classic/LinAlg/libtpetraclassiclinalg.so.13.0 ../../tpetra/classic/NodeAPI/libtpetraclassicnodeapi.so.13.0 ../../tpetra/classic/src/libtpetraclassic.so\
.13.0 ../../teuchos/kokkoscomm/src/libteuchoskokkoscomm.so.13.0 ../../teuchos/kokkoscompat/src/libteuchoskokkoscompat.so.13.0 ../../teuchos/remainder/src/libteucho\
sremainder.so.13.0 ../../teuchos/numerics/src/libteuchosnumerics.so.13.0 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lmkl_intel_lp64 -lmkl_sequential -\
lmkl_core -lpthread ../../teuchos/comm/src/libteuchoscomm.so.13.0 ../../teuchos/parameterlist/src/libteuchosparameterlist.so.13.0 ../../teuchos/parser/src/libteuch\
osparser.so.13.0 ../../teuchos/core/src/libteuchoscore.so.13.0 ../../kokkos/core/src/libkokkoscore.so.3.1.1 /usr/lib64/libdl.so
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../src/libamesos.so.13.0: undefined reference to `blacs_gridinfo_'
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../src/libamesos.so.13.0: undefined reference to `pdpotrf_'

As you see, missing BLACS/Scalapack routines and the mkl_scalapack library is not linked in.

jhux2 commented 2 years ago

@bartlettroscoe According to the documentation in the subsection TPL Example 2: Intel Math Kernel Library (MKL) for BLAS, cmake should prepend the value of BLAS_LIBRARY_DIRS to each of the libraries in BLAS_LIBRARY_NAMES, but this isn't happening.

For example, the Intel Math Kernel Library (MKL) implementation for the BLAS is usually given in several libraries. The exact set of libraries needed depends on the version of MKL, whether 32bit or 64bit libraries are needed, etc. Figuring out the correct set and ordering of these libraries for a given platform may not be trivial. But once the set and the order of the libraries is known, then one can provide the correct list at configure time.

For example, suppose one wants to use the threaded MKL libraries listed in the directories:

/usr/local/intel/Compiler/11.1/064/mkl/lib/em64t/
/usr/local/intel/Compiler/11.1/064/lib/intel64/

and the list of libraries being searched for is mkl_intel_lp64, mkl_intel_thread, mkl_core and iomp5.

In this case, one could specify this with the following do-configure script:

#!/bin/bash

INTEL_DIR=/usr/local/intel/Compiler/11.1/064

cmake \
  -D TPL_ENABLE_BLAS=ON \
  -D BLAS_LIBRARY_DIRS="${INTEL_DIR}/em64t;${INTEL_DIR}/intel64" \
  -D BLAS_LIBRARY_NAMES="mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5" \
  ...
  ${PROJECT_SOURCE_DIR}

This would call find_library() on each of the listed library names in these directories and would find them and list them in:

-- TPL_BLAS_LIBRARIES='/usr/local/intel/Compiler/11.1/064/em64t/libmkl_intel_lp64.so;...'

Do you or any @trilinos/amesos2 developer have idea what could be going wrong? Here is a snippet from @VictorEijkhout's configure line.

-D BLAS_LIBRARY_DIRS:PATH=/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin
-D 'BLAS_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64'
VictorEijkhout commented 2 years ago

Maybe the intel compiler structure has changed between my previous build and the current. Those people just can't make up their mind about directory structure. I did find that cmake couldn't find one of the mkl libraries, but then installation went on regardless. I think you should abort if there are obvious problems with the configuration.

bartlettroscoe commented 2 years ago

cmake should prepend the value of BLAS_LIBRARY_DIRS to each of the libraries in BLAS_LIBRARY_NAMES, but this isn't happening

That is not what happens. What happens is that the directories listed under <TPLNAME>_LIBRARY_DIRS are searched for the libraries listed in <TPLNAME>_LIBRARY_NAMES. From:

it says:

<TPLNAME>_LIBRARY_DIRS:PATH: The list of directories where the library files will be searched for using find_library(), for each library, in order.

If you look at the output from cmake, you can see that is what is happening.

If you want to be precise about what libraries are used from what directories, take the bull by the horns and just list the full set of libraries in order with their full paths by setting TPL_<tplName>_LIBRARIES. This is what the ATDM Trilinos builds due for MLK (when they don't use the -mkl option with the Intel compiler). Here is the bash code that constructs that full list of MKL libraries for some of the ATDM Trilinos builds:

https://github.com/trilinos/Trilinos/blob/7bc4eb697c4ccd9fab1742661da3260a1069d41a/cmake/std/atdm/cee-rhel7/environment.sh#L215-L225

with the implementation at:

https://github.com/trilinos/Trilinos/blob/7bc4eb697c4ccd9fab1742661da3260a1069d41a/cmake/std/atdm/utils/atdm_config_helper_funcs.sh#L1-L20

bartlettroscoe commented 2 years ago

I did find that cmake couldn't find one of the mkl libraries, but then installation went on regardless. I think you should abort if there are obvious problems with the configuration.

Well that is likely bad. If you want it to abort on a failure to find, then say:

    -D Trilinos_MUST_FIND_ALL_TPL_LIBS=TRUE

See:

(This could not be made the default because it breaks backward compatibility. But with Trilinos 14.0 coming up, we could set that to ON by default.)

bartlettroscoe commented 2 years ago

@VictorEijkhout, if you want help debugging this, attach:

  1. The configure script that runs cmake
  2. The full STDOUT+STDERR from running cmake
  3. The full link line that fails.

Also, I would advice you to use Ninja. Then you can just look at the build.ninja file to see the full list of link libraries for every target.

bartlettroscoe commented 2 years ago

@VictorEijkhout

Browsing above, this configuration looks to have all kinds of issues. We need to start with the input to CMake and the STDOUT+STDOUT from cmake and the CMakeCache.txt file. And if you use Ninja, the build.ninja file would be nice too. (Seems there is as 25MB limit for these types of files so GitHub should allow it.)

And, of course, we need to know the required list and order the libraries should be appearing on the link lines. (We can not determine that for you because we are not on the system.)

VictorEijkhout commented 2 years ago 
-D Trilinos_MUST_FIND_ALL_TPL_LIBS=TRUE

That sounds eminently useful. Trying now.

VictorEijkhout commented 2 years ago

attach:

Really? Why doesn't your build system collect that information? Why don't you have a nice system like PETSc where users are told to send configure.log and make.log and that's enough for debugging? How do you live with a system that doesn't log its own workings, especially given how complicated the interactions in Trilinos are?

Btw, I don't install interactively: I write a spec file for rpmbuild because our cluster setup uses rpms for installation. I'll take a look at ninja but I have my doubts that it would make life any easier for me.

VictorEijkhout commented 2 years ago 
# ls -l /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_scalapack_lp64*
-rw-r--r-- 1 root root 12306444 Feb  8  2020 /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_scalapack_lp64.a
-rwxr-xr-x 1 root root  7735678 Feb  8  2020 /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_scalapack_lp64.so

but

Processing enabled TPL: MPI (enabled explicitly, disable with -DTPL_ENABLE_MPI=OFF)
Processing enabled TPL: BLAS (enabled by TeuchosNumerics, disable with -DTPL_ENABLE_BLAS=OFF)
-- BLAS_LIBRARY_NAMES='mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64'
-- Must find at least one lib in each of the lib sets "mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64"
-- Searching for libs in BLAS_LIBRARY_DIRS='/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin'
-- Searching for a lib in the set "mkl_intel_lp64":
--   Searching for lib 'mkl_intel_lp64' ...
--     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so'
-- Searching for a lib in the set "mkl_sequential":
--   Searching for lib 'mkl_sequential' ...
--     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_sequential.so'
-- Searching for a lib in the set "mkl_core":
--   Searching for lib 'mkl_core' ...
--     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_core.so'
-- Searching for a lib in the set "pthread":
--   Searching for lib 'pthread' ...
--     Found lib '/usr/lib64/libpthread.so'
-- Searching for a lib in the set "libmkl_scalapack_lp64":
--   Searching for lib 'libmkl_scalapack_lp64' ...
-- ERROR: Did not find a lib in the lib set "libmkl_scalapack_lp64" for the TPL 'BLAS'!
bartlettroscoe commented 2 years ago

Really? Why doesn't your build system collect that information? Why don't you have a nice system like PETSc where users are told to send configure.log and make.log and that's enough for debugging? How do you live with a system that doesn't log its own workings, especially given how complicated the interactions in Trilinos are?

What system are you working with that does not log the CMake configure output? For example, CMake ExternalProject logs each command that it runs so it would be easy to grab that log and send it. Any system from driving CMake builds that does not log the cmake STDOUT is not a very good system for driving CMake projects. (I guess you can debate the wisdom of using CMake but I think that ship has sailed for the CSE and HPC communities; with some notable stragglers.)

VictorEijkhout commented 2 years ago

For example, CMake ExternalProject logs

Then why didn't you ask me to send that one file?

bartlettroscoe commented 2 years ago

Then why didn't you ask me to send that one file?

@VictorEijkhout, that only applies if you are using CMake ExternalProject to drive builds. It sounds like you are not. I was just using CMake ExternalProject as an example of a package build orchestration system that logs the CMake configure. Note that Spack also creates log files for each step of the installation process as well.

Does rpmbuild not log the various steps in doing a package build and install including the cmake configure command?

VictorEijkhout commented 2 years ago

that only applies if you are using CMake ExternalProject to drive builds

That seems to be a module, meaning something that is used in cmake configuration files, right? So you'd have to put that in your scripts. Or can I add this as a commandline option to cmake?

VictorEijkhout commented 2 years ago

Does rpmbuild not log

That's the 20M stdout that I was sending you. But that's really beside the point. rpmbuild runs a script, and in that script I put the cmake and make calls and whatever. If your cmake fails, I shouldn't send you a script that contains the irrelevant stuff like untarring, copying, module loading, environment variable setting.

bartlettroscoe commented 2 years ago

that only applies if you are using CMake ExternalProject to drive builds

That seems to be a module, meaning something that is used in cmake configuration files, right?

@VictorEijkhout, CMake ExternalProject is just a "package dependency handling and build/install/test orchestration tool" (See slide 3 of https://doi.org/10.6084/m9.figshare.19241715). Think of CMake ExternalProject as an alternative implementation of Spack, except it is 100% implemented in the CMake language instead of in Python.

The command rpmbuild is just used as part of the RPM implementation of a "package dependency handling and build/install orchestration tool".

bartlettroscoe commented 2 years ago

If your cmake fails, I shouldn't send you a script that contains the irrelevant stuff like untarring, copying, module loading, environment variable setting.

Then just send the part of the rpmbuild log file that shows the STDOUT/STDERR for the cmake command along with the command-line arguments used to call CMake. I am sure it is there.

VictorEijkhout commented 2 years ago

just send the part

That's the 20M log I sent you. Why doesn't cmake save its thinking in a file tthat I can send you?

VictorEijkhout commented 2 years ago

Meanwhile I'm stuck as per the above on cmake not finding a file that is definitely there.

bartlettroscoe commented 2 years ago

That's the 20M log I sent you.

Where?

Why doesn't cmake save its thinking in a file tthat I can send you?

I agree, that would be a great CMake feature. I don't see any sign this feature already exists from looking at:

It seems like it should be pretty easy to create a log file that contains the CMake command-line options, the initial cache vars and the STDOUT+STDERR in one file (or perhaps as set of files). With C++ ostsream (and CMake is implemented in C++), it is easy to create a splitting subclass that could do send to both STDOUT+STDERR and write to a file buffer in a slick way.

You could post a feature request to Kitware to ask for this at:

?

There may already be a request for this but I can't find one looking at:

(but that brings up 380 issues so it could be there).

But CMake also creates a bunch of other log files, like for compiler checks, try_compile() commands, etc. To collect the entire log of everything interesting to debug would be a lot of files form all over the binary directory tree.

When you have been working with a given tool or approach for long enough, you forgot what the tool does not support since you have been using workarounds and accommodations for many years. I certainly suffer from this with CMake.

Meanwhile I'm stuck as per the above on cmake not finding a file that is definitely there.

The cmake STDOUT/STDERR will help to show this and then we can go from there.

VictorEijkhout commented 2 years ago

he cmake STDOUT/STDERR will help to show this and then we can go from there.

That's what I pasted. For good measure again:

# ls -l /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_scalapack_lp64*
-rw-r--r-- 1 root root 12306444 Feb  8  2020 /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_scalapack_lp64.a
-rwxr-xr-x 1 root root  7735678 Feb  8  2020 /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_scalapack_lp64.so

but

Processing enabled TPL: MPI (enabled explicitly, disable with -DTPL_ENABLE_MPI=OFF)
Processing enabled TPL: BLAS (enabled by TeuchosNumerics, disable with -DTPL_ENABLE_BLAS=OFF)
-- BLAS_LIBRARY_NAMES='mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64'
-- Must find at least one lib in each of the lib sets "mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64"
-- Searching for libs in BLAS_LIBRARY_DIRS='/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin'
-- Searching for a lib in the set "mkl_intel_lp64":
--   Searching for lib 'mkl_intel_lp64' ...
--     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so'
-- Searching for a lib in the set "mkl_sequential":
--   Searching for lib 'mkl_sequential' ...
--     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_sequential.so'
-- Searching for a lib in the set "mkl_core":
--   Searching for lib 'mkl_core' ...
--     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_core.so'
-- Searching for a lib in the set "pthread":
--   Searching for lib 'pthread' ...
--     Found lib '/usr/lib64/libpthread.so'
-- Searching for a lib in the set "libmkl_scalapack_lp64":
--   Searching for lib 'libmkl_scalapack_lp64' ...
-- ERROR: Did not find a lib in the lib set "libmkl_scalapack_lp64" for the TPL 'BLAS'!
bartlettroscoe commented 2 years ago

The cmake STDOUT/STDERR will help to show this and then we can go from there.

That's what I pasted.

You posted the full cmake STDOUT+STDERR? I need to see the full output, not just selected lines.

...
-- Searching for a lib in the set "libmkl_scalapack_lp64":
--   Searching for lib 'libmkl_scalapack_lp64' ...
-- ERROR: Did not find a lib in the lib set "libmkl_scalapack_lp64" for the TPL 'BLAS'!

Well that is bad obviously. I have seen bugs in some versions of CMake with find commands. What version of CMake are you using here? (I would already know that if the full cmake STDOUT+STDERR was attached.)

Before go any further, please, post the full:

  1. The configure script that runs cmake
  2. The full STDOUT+STDERR from running cmake
jhux2 commented 2 years ago 
# ls -l /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_scalapack_lp64*
-rw-r--r-- 1 root root 12306444 Feb  8  2020 /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_scalapack_lp64.a
-rwxr-xr-x 1 root root  7735678 Feb  8  2020 /opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_scalapack_lp64.so

but

Processing enabled TPL: MPI (enabled explicitly, disable with -DTPL_ENABLE_MPI=OFF)
Processing enabled TPL: BLAS (enabled by TeuchosNumerics, disable with -DTPL_ENABLE_BLAS=OFF)
-- BLAS_LIBRARY_NAMES='mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64'
-- Must find at least one lib in each of the lib sets "mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64"
-- Searching for libs in BLAS_LIBRARY_DIRS='/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin'
-- Searching for a lib in the set "mkl_intel_lp64":
--   Searching for lib 'mkl_intel_lp64' ...
--     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so'
-- Searching for a lib in the set "mkl_sequential":
--   Searching for lib 'mkl_sequential' ...
--     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_sequential.so'
-- Searching for a lib in the set "mkl_core":
--   Searching for lib 'mkl_core' ...
--     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_core.so'
-- Searching for a lib in the set "pthread":
--   Searching for lib 'pthread' ...
--     Found lib '/usr/lib64/libpthread.so'
-- Searching for a lib in the set "libmkl_scalapack_lp64":
--   Searching for lib 'libmkl_scalapack_lp64' ...
-- ERROR: Did not find a lib in the lib set "libmkl_scalapack_lp64" for the TPL 'BLAS'!

@VictorEijkhout The search is prepending a lib to each library specified. For example,

 --   Searching for lib 'mkl_sequential' ...
 --     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_sequential.so'

I wonder if the search function is getting tripped up by the fact that the last two libraries libmkl_scalapack_lp64 and `libmkl_blacs_intelmpi_lp64' already have lib in their name.

bartlettroscoe commented 2 years ago

The search is prepending a lib to each library specified

Bingo!

@VictorEijkhout, for

BLAS_LIBRARY_NAMES='mkl_intel_lp64;mkl_sequential;mkl_core;pthread;libmkl_scalapack_lp64;libmkl_blacs_intelmpi_lp64

you need to replace libmkl_blacs_intelmpi_lp64 with mkl_blacs_intelmpi_lp64.

Also, I strongly suspect this is not a valid library link order. My guess it that something more like:

BLAS_LIBRARY_NAMES='mkl_scalapack_lp64;mkl_blacs_intelmpi_lp64;mkl_intel_lp64;mkl_sequential;mkl_core;pthread

might be the correct link library order.

The problem is that you don't pass libmkl_scalapack_lp64 in to find_libray() to find libmkl_scalapack_lp64.so, you need to pass in mkl_scalapack_lp64. CMake automatically adds the prefix and suffix to the name for different platforms. See:

which mentions how when looking for <name> on Linux, CMake will automatically look for lib<name>.so and lib<name>.a. (And on Windows, it will look for <name>.lib or <name>.dll.a, for example.)

As @jhux2 mentioned above, that should have been the clue when it showed, for example:

--   Searching for lib 'mkl_intel_lp64' ...
--     Found lib '/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so'

See, it was looking for mkl_intel_lp64 and find_library() found libmkl_intel_lp64.so.

Note that if you give the full name libmkl_scalapack_lp64.so, then find_libray() would actually find that file (but you almost never want to do that so that you have flexibility to find shared or static libs).

Note that the documentation does try to make these it clear that find_file() and find_library() are used internally and how to pass in library names correctly at:

which says:

  • <TPLNAME>_INCLUDE_DIRS:PATH: List of paths to search for header files using find_file() for each header file, in order.
  • <TPLNAME>_LIBRARY_NAMES:STRING: List of unadorned library names, in the order of the link line. The platform-specific prefixes (e.g.. 'lib') and postfixes (e.g. '.a', '.lib', or '.dll') will be added automatically by CMake. For example, the library libblas.so, libblas.a, blas.lib or blas.dll will all be found on the proper platform using the name blas.
  • <TPLNAME>_LIBRARY_DIRS:PATH: The list of directories where the library files will be searched for using find_library(), for each library, in order.

Does that make sense?

jhux2 commented 2 years ago

@VictorEijkhout Is this resolved?

VictorEijkhout commented 2 years ago

I keep running into this sort of thing:

/admin/build/admin/rpms/stampede2/BUILD/trilinos-13.0.1/packages/kokkos-kernels/src/sparse/impl/KokkosSparse_spmv_impl.hpp(791): remark #15009: _ZNK12KokkosSparse4Im\
pl26SPMV_MV_LayoutLeft_FunctorINS_9CrsMatrixIKdKiN6Kokkos6DeviceINS5_6SerialENS5_9HostSpaceEEENS5_12MemoryTraitsILj1EEEKmEENS5_4ViewIPPS3_JNS5_10LayoutLeftES9_NSA_IL\
j3EEEEEENSE_IPPdJSH_S9_SB_EEELin1ELi1ELb0EEclERKNS5_4Impl20HostThreadTeamMemberIS7_EE has been targeted for automatic cpu dispatch
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `Pa\
rMETIS_V3_NodeND'
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `ME\
TIS_PartGraphKway'
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `me\
tis_setdefaultoptions_'
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `ME\
TIS_SetDefaultOptions'
/opt/apps/gcc/9.1.0/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: ../../../../../amesos/src/libamesos.so.13.0: undefined reference to `me\
tis_nodend_'

This must be caused by the fact that petsc is built with a ton of external libraries. Where do I add these to a cmake variable?

VictorEijkhout commented 2 years ago

I will parse the above comment about STUFF_LIBRARY_NAMES.

VictorEijkhout commented 2 years ago

Oh wait, Amesos is not a TPL, right? So how do I specify extra libraries for Amesos?

  1. Btw, shouldn't cmake test if I specified enough libraries?

  2. My build file has grown organically over the years and I suspect I have many options that are not actually used. Where does cmake report such options?

github-actions[bot] commented 1 year ago

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE. If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

github-actions[bot] commented 1 year ago

This issue was closed due to inactivity for 395 days.