SuiteSparse/KLU: openmp flags do not propagate

[VictorEijkhout]

[  8%] Building C object packages/seacas/libraries/chaco/CMakeFiles/chaco.dir/internal/force_internal.c.o
cd /work2/00434/eijkhout/trilinos/build-14.0.0-clx-intel23-impi/packages/seacas/libraries/chaco && /opt/intel/oneapi/mpi/2021.9.0/bin/mpicc -Dchaco_EXPORTS -I/work2/00434/eijkhout/trilinos/build-1
4.0.0-clx-intel23-impi -I/work2/00434/eijkhout/trilinos/trilinos-14.0.0/packages/seacas/libraries/chaco/main -I/work2/00434/eijkhout/trilinos/trilinos-14.0.0/packages/seacas/libraries/chaco/util -
fiopenmp -O3 -DNDEBUG -fPIC -MD -MT packages/seacas/libraries/chaco/CMakeFiles/chaco.dir/internal/force_internal.c.o -MF CMakeFiles/chaco.dir/internal/force_internal.c.o.d -o CMakeFiles/chaco.dir/
internal/force_internal.c.o -c /work2/00434/eijkhout/trilinos/trilinos-14.0.0/packages/seacas/libraries/chaco/internal/force_internal.c
/work2/00434/eijkhout/trilinos/trilinos-14.0.0/packages/common/auxiliarySoftware/SuiteSparse/src/KLU/Source/trilinos_klu_factor.c:506:19: error: call to undeclared library function 'omp_get_num_threads' with type 'int (void)'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    num_threads = omp_get_num_threads();
                  ^
/work2/00434/eijkhout/trilinos/trilinos-14.0.0/packages/common/auxiliarySoftware/SuiteSparse/src/KLU/Source/trilinos_klu_factor.c:506:19: note: include the header <omp.h> or explicitly provide a declaration for 'omp_get_num_threads'
cd /work2/00434/eijkhout/trilinos/build-14.0.0-clx-intel23-impi/packages/shards/src && /opt/apps/cmake/3.24.2/bin/cmake -E cmake_symlink_library libshards.so.14.0 libshards.so.14 libshards.so
1 error generated.
[  8%] Building C object packages/seacas/libraries/chaco/CMakeFiles/chaco.dir/internal/improve_internal.c.o
make[3]: *** [packages/common/auxiliarySoftware/SuiteSparse/src/CMakeFiles/trilinosss.dir/KLU/Source/trilinos_klu_factor.c.o] Error 1

But:

++ cmake -B /work2/00434/eijkhout/trilinos/build-14.0.0-clx-intel23-impi -S /work2/00434/eijkhout/trilinos/trilinos-14.0.0 -D CMAKE_INSTALL_PREFIX:PATH=/work2/00434/eijkhout/trilinos/installation-14.0.0-clx-intel23-impi -D CMAKE_BUILD_TYPE:STRING=RELEASE -D BUILD_SHARED_LIBS:BOOL=ON -D Trilinos_VERBOSE_CONFIGURE=ON -D CMAKE_VERBOSE_MAKEFILE=ON -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF -D Trilinos_DUMP_PACKAGE_DEPENDENCIES=ON -D Trilinos_ASSERT_MISSING_PACKAGES=OFF -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF -D Trilinos_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_EXAMPLES:BOOL=ON -D Trilinos_ENABLE_Export_Makefiles:BOOL=ON -D Trilinos_ENABLE_Fortran:BOOL=ON -D CMAKE_C_COMPILER=mpicc -D CMAKE_CXX_COMPILER=mpicxx -D CMAKE_Fortran_COMPILER=mpif90 -D CMAKE_CXX_STANDARD:STRING=17 -D Trilinos_ENABLE_OpenMP:BOOL=ON -D TPL_ENABLE_MPI:BOOL=ON -D MPI_BASE_DIR=/opt/intel/oneapi/mpi/2021.9.0/intel64 -D MPI_INCLUDE_DIRS=/opt/intel/oneapi/mpi/2021.9.0/include -D MPI_EXEC:FILEPATH=/opt/apps/xalt/0.6/bin/ibrun -D TPL_ENABLE_GLM=OFF -D TPL_ENABLE_Matio=OFF -D TPL_ENABLE_BLAS=ON -D BLAS_LIBRARY_DIRS:PATH=/opt/intel/oneapi/mkl/2023.1.0/lib/intel64 -D 'BLAS_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5;pthread' -D BLAS_INCLUDE_DIRS:PATH=/opt/intel/oneapi/mkl/2023.1.0/include -D LAPACK_INCLUDE_DIRS:PATH=/opt/intel/oneapi/mkl/2023.1.0/include -D LAPACK_LIBRARY_DIRS:PATH=/opt/intel/oneapi/mkl/2023.1.0/lib/intel64 -D 'LAPACK_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;iomp5;pthread' -D TPL_ENABLE_HDF5:BOOL=ON -D HDF5_INCLUDE_DIRS:PATH=/work2/00434/eijkhout/hdf5/installation-1.14-clx-intel23-impi/include -D HDF5_LIBRARY_DIRS:PATH=/work2/00434/eijkhout/hdf5/installation-1.14-clx-intel23-impi/lib -D TPL_ENABLE_Netcdf:BOOL=ON -D Netcdf_INCLUDE_DIRS:PATH=/work2/00434/eijkhout/netcdf/installation-4.9.2-clx-intel23-impi/include -D Netcdf_LIBRARY_DIRS:PATH=/work2/00434/eijkhout/netcdf/installation-4.9.2-clx-intel23-impi
/lib -D Tpetra_INST_DOUBLE:BOOL=ON -D Tpetra_INST_FLOAT:BOOL=OFF -D Tpetra_INST_COMPLEX_FLOAT:BOOL=OFF -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF -D Tpetra_INST_INT_LONG:BOOL=OFF -D Tpetra_INST_INT_UN
SIGNED:BOOL=OFF -D TPL_ENABLE_Boost:BOOL=ON -D Boost_INCLUDE_DIRS:PATH=/work2/00434/eijkhout/boost/installation-1.81.0-clx-intel23/include -D Boost_LIBRARY_DIRS:PATH=/work2/00434/eijkhout/boost/in
stallation-1.81.0-clx-intel23/lib -D TPL_ENABLE_BoostLib:BOOL=ON -D BoostLib_INCLUDE_DIRS:PATH=/work2/00434/eijkhout/boost/installation-1.81.0-clx-intel23/include -D BoostLib_LIBRARY_DIRS:PATH=/wo
rk2/00434/eijkhout/boost/installation-1.81.0-clx-intel23/lib -D TPL_ENABLE_MUMPS:BOOL=0 -D MUMPS_INCLUDE_DIRS= -D MUMPS_LIBRARY_DIRS= -D MUMPS_LIBRARY_NAMES:STRING= -D VLE_TPL_MUMPS_LIBRARIES:STRI
NG= -D TPL_ParMETIS_LIBRARIES= -D TPL_ParMETIS_INCLUDE_DIRS= -D TPL_ENABLE_yaml-cpp:BOOL=0 -D yaml-cpp_INCLUDE_DIRS:PATH= -D yaml-cpp_LIBRARY_DIRS:PATH= -D Trilinos_ENABLE_Amesos:BOOL=ON -D Trilin
os_ENABLE_Amesos2:BOOL=ON -D Amesos2_ENABLE_KLU2:BOOL=ON -D Amesos2_ENABLE_Basker:BOOL=ON -D Trilinos_ENABLE_Anasazi:BOOL=ON -D Trilinos_ENABLE_AztecOO:Bool=ON -D Trilinos_ENABLE_Belos:BOOL=ON -D
Trilinos_ENABLE_Epetra:Bool=ON -D Trilinos_ENABLE_EpetraExt:Bool=ON -D Epetra_ENABLE_TESTS:BOOL=ON -D Trilinos_ENABLE_ExodusII:Bool=ON -D Trilinos_ENABLE_FEI:Bool=ON -D Trilinos_ENABLE_Ifpack:Bool
=ON -D Trilinos_ENABLE_Ifpack2:BOOL=ON -D Trilinos_ENABLE_Intrepid:BOOL=ON -D Trilinos_ENABLE_Intrepid2:BOOL=ON -D Intrepid_ENABLE_TESTS:BOOL=ON -D Trilinos_ENABLE_Isorropia:BOOL=ON -D Trilinos_EN
ABLE_ML:BOOL=ON -D ML_TAKES_SUPERLU_LESS_THAN_5=TRUE -D ML_ENABLE_SuperLU:BOOL=OFF -D Trilinos_ENABLE_MOOCHO:BOOL=ON -D Trilinos_ENABLE_MueLu:BOOL= -D MueLu_ENABLE_Tutorial:BOOL=OFF -D MueLu_ENABL
E_EXAMPLES:BOOL=OFF -D Trilinos_ENABLE_NOX=ON -D NOX_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_Pamgen:Bool=ON -D Trilinos_ENABLE_Panzer:Bool=ON -D Trilinos_ENABLE_Phalanx:BOOL=ON -D Phalanx_EXPLICI
T_TEMPLATE_INSTANTIATION=ON -D Phalanx_ENABLE_EXAMPLES=OFF -D Trilinos_ENABLE_Piro:BOOL=ON -D Trilinos_ENABLE_Rythmos:BOOL=ON -D Trilinos_ENABLE_Sacado:Bool=ON -D Trilinos_ENABLE_SEACAS:BOOL=ON -D
 Trilinos_ENABLE_SEACASIoss:BOOL=ON -D Trilinos_ENABLE_SEACASBlot:BOOL=ON -D Trilinos_ENABLE_SEACASExodus:BOOL=ON -D Trilinos_ENABLE_SECONDARY_STABLE_CODE:BOOL=ON -D Trilinos_ENABLE_Shards:BOOL=ON
 -D Trilinos_ENABLE_ShyLU:BOOL=OFF -D Trilinos_ENABLE_Stokhos:BOOL=ON -D Trilinos_ENABLE_Stratimikos:BOOL=ON -D Trilinos_ENABLE_Teko:BOOL=ON -D Trilinos_ENABLE_Teuchos:BOOL=ON -D Teuchos_ENABLE_LO
NG_LONG_INT:BOOL=ON -D Trilinos_ENABLE_Thyra:BOOL=ON -D Trilinos_ENABLE_Tpetra:BOOL=ON -D Trilinos_ENABLE_TriKota:BOOL=ON -D Trilinos_ENABLE_Zoltan:BOOL=ON -D Trilinos_ENABLE_Zoltan2:BOOL=ON -D KO
KKOS_IS_REQUIRED_FOR_SEVERAL_PACKAGES=off -D Trilinos_ENABLE_Kokkos:BOOL=ON -D Trilinos_ENABLE_KokkosCore:BOOL=ON -D Phalanx_KOKKOS_DEVICE_TYPE:STRING=OPENMP -D Phalanx_INDEX_SIZE_TYPE:STRING=INT
-D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF -D Kokkos_ENABLE_Serial:BOOL=ON -D Kokkos_ENABLE_OpenMP:BOOL=ON -D Kokkos_ENABLE_Pthread:BOOL=OFF -D Trilinos_ENABLE_STK:BOOL=0 -D Trilinos_ENABLE_STKI
O:BOOL=0 -D Trilinos_ENABLE_STKMesh:BOOL=0 -D SWIG_EXECUTABLE:FILEPATH=/work2/00434/eijkhout/swig/installation-4.1.1-clx-intel23/bin/swig

[VictorEijkhout]

logs trilinos_logs.zip

[jhux2]

@srajama1 Any ideas who should look at this?

[VictorEijkhout]

Hm?

[jhux2]

@trilinos/amesos2 @ccober6

[jhux2]

I don't know that this is an Amesos2 issue, just not sure who to ping here

[ccober6]

@trilinos/framework Can you take a look at this and do some initial triage?

[VictorEijkhout]

As a stopgap I'll take a hint how to disable the packages that trigger this. You have my cmake line above.

[VictorEijkhout]

It's been another week.

[brian-kelley]

@trilinos/amesos2 -fiopenmp is getting passed to the compiler, but I don't see being included anywhere. I haven't replicated this issue but the fix might just be:

diff --git a/packages/common/auxiliarySoftware/SuiteSparse/src/KLU/Include/trilinos_klu_internal.h b/packages/common/auxiliarySoftware/SuiteSparse/src/KLU/Include/trilinos_klu_internal.h
index 49d25004330..5d26db1f549 100644
--- a/packages/common/auxiliarySoftware/SuiteSparse/src/KLU/Include/trilinos_klu_internal.h
+++ b/packages/common/auxiliarySoftware/SuiteSparse/src/KLU/Include/trilinos_klu_internal.h
@@ -37,6 +37,10 @@
 #include <stdlib.h>
 #include <math.h>

+#ifdef TRILINOSSS_HAVE_OMP
+#include <omp.h>
+#endif
+
 #undef ASSERT
 #ifndef NDEBUG
 #define ASSERT(a) assert(a)

[VictorEijkhout]

That seems to do the trick. If you're about to make maintainance release of both 13 and 14.....

[ccober6]

@VictorEijkhout can you submit a PR with the suggested changes?

[VictorEijkhout]

Does that help in also getting an update to version 13? I still need that for DealII.

[brian-kelley]

@VictorEijkhout Glad to hear this fixed the issue. I went ahead and opened #11949 with that patch so it will at least be fixed in develop and future releases.