Panzer: configure flag in phalanx can allow for inconsistent execution space between packages

[VictorEijkhout]

/work2/00434/eijkhout/trilinos/trilinos-13.4.1/packages/kokkos/core/src/Kokkos_Parallel.hpp(96): error: static assertion failed with "A policy with an execution space and a functor with an execu
tion space are given but the execution space types do not match!"
    static_assert(
    ^
          detected during:
            instantiation of class "Kokkos::Impl::FunctorPolicyExecutionSpace<Functor, Policy> [with Functor=panzer::Integrator_BasisTimesVector<panzer::Traits::Jacobian, panzer::Traits>, Policy
=Kokkos::RangePolicy<panzer::Integrator_BasisTimesVector<panzer::Traits::Jacobian, panzer::Traits>::FieldMultTag<0>>]" at line 337 of "/work2/00434/eijkhout/trilinos/trilinos-13.4.1/packages/kok
kos/core/src/Kokkos_Core_fwd.hpp"
            processing of template argument list for "Kokkos::Impl::ParallelFor" based on template arguments <panzer::Integrator_BasisTimesVector<panzer::Traits::Jacobian, panzer::Traits>, Kokko
s::RangePolicy<panzer::Integrator_BasisTimesVector<panzer::Traits::Jacobian, panzer::Traits>::FieldMultTag<0>>> at line 166
            instantiation of "void Kokkos::parallel_for(const ExecPolicy &, const FunctorType &, const std::string &, std::enable_if<Kokkos::is_execution_policy<ExecPolicy>::value, void>::type *
) [with ExecPolicy=Kokkos::RangePolicy<panzer::Integrator_BasisTimesVector<panzer::Traits::Jacobian, panzer::Traits>::FieldMultTag<0>>, FunctorType=panzer::Integrator_BasisTimesVector<panzer::Tr
aits::Jacobian, panzer::Traits>]" at line 387 of
                      "/work2/00434/eijkhout/trilinos/trilinos-13.4.1/packages/panzer/disc-fe/src/evaluators/Panzer_Integrator_BasisTimesVector_impl.hpp"
            instantiation of "void panzer::Integrator_BasisTimesVector<EvalT, Traits>::evaluateFields(Traits::EvalData) [with EvalT=panzer::Traits::Jacobian, Traits=panzer::Traits]" at line 49 o
f "/work2/00434/eijkhout/trilinos/trilinos-13.4.1/packages/panzer/disc-fe/src/evaluators/Panzer_Integrator_BasisTimesVector.cpp"

++ cmake -Wno-dev -D CMAKE_INSTALL_PREFIX:PATH=/work2/00434/eijkhout/trilinos/installation-13.4.1-clx-intel-impi -D CMAKE_BUILD_TYPE:STRING=RELEASE -D BUILD_SHARED_LIBS:BOOL=ON -D Trilinos_VERBO
SE_CONFIGURE=ON -D CMAKE_VERBOSE_MAKEFILE=ON -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF -D Trilinos_ASSERT_MISSING_PACKAGES=OFF -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF -D Trilinos_ENABLE
_TESTS:BOOL=OFF -D Trilinos_ENABLE_EXAMPLES:BOOL=ON -D Trilinos_ENABLE_Export_Makefiles:BOOL=ON -D Trilinos_ENABLE_Fortran:BOOL=ON -D CMAKE_C_COMPILER=mpicc -D CMAKE_CXX_COMPILER=mpicxx -D CMAKE
_Fortran_COMPILER=mpif90 -D CMAKE_CXX_STANDARD:STRING=17 -D Trilinos_EXTRA_LINK_FLAGS= -D TPL_ENABLE_MPI:BOOL=ON -D Trilinos_ENABLE_OpenMP=ON -D MPI_BASE_DIR=/intel64 -D MPI_INCLUDE_DIRS= -D MPI
_EXEC:FILEPATH=/opt/apps/xalt/0.6/bin/ibrun -D TPL_MPI_BASE_DIR=intel64 -D TPL_MPI_INCLUDE_DIRS= -D TPL_ENABLE_GLM=OFF -D TPL_ENABLE_Matio=OFF -D TPL_ENABLE_BLAS=ON -D BLAS_LIBRARY_DIRS:PATH=/sc
ratch1/projects/compilers/oneapi_2022.3.0/mkl/latest/lib/intel64 -D 'BLAS_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_intel_thread;mkl_core;iomp5;pthread' -D BLAS_INCLUDE_DIRS:PATH=/scratch1/project
s/compilers/oneapi_2022.3.0/mkl/latest/include -D LAPACK_INCLUDE_DIRS:PATH=/scratch1/projects/compilers/oneapi_2022.3.0/mkl/latest/include -D LAPACK_LIBRARY_DIRS:PATH=/scratch1/projects/compiler
s/oneapi_2022.3.0/mkl/latest/lib/intel64 -D 'LAPACK_LIBRARY_NAMES:STRING=mkl_intel_lp64;mkl_sequential;mkl_core;iomp5;pthread' -D TPL_ENABLE_HDF5:BOOL=0 -D HDF5_INCLUDE_DIRS:PATH= -D HDF5_LIBRAR
Y_DIRS:PATH= -D TPL_ENABLE_Netcdf:BOOL=0 -D Netcdf_INCLUDE_DIRS:PATH= -D Netcdf_LIBRARY_DIRS:PATH= -D Tpetra_INST_DOUBLE:BOOL=ON -D Tpetra_INST_FLOAT:BOOL=OFF -D Tpetra_INST_COMPLEX_FLOAT:BOOL=O
FF -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=OFF -D Tpetra_INST_INT_LONG:BOOL=OFF -D Tpetra_INST_INT_UNSIGNED:BOOL=OFF ' ' -D TPL_ENABLE_Boost:BOOL=ON -D Boost_INCLUDE_DIRS:PATH=/work2/00434/eijkhout/b
oost/installation-1.81.0-clx-intel/include -D Boost_LIBRARY_DIRS:PATH=/work2/00434/eijkhout/boost/installation-1.81.0-clx-intel/lib -D TPL_ENABLE_BoostLib:BOOL=ON -D BoostLib_INCLUDE_DIRS:PATH=/
work2/00434/eijkhout/boost/installation-1.81.0-clx-intel/include -D BoostLib_LIBRARY_DIRS:PATH=/work2/00434/eijkhout/boost/installation-1.81.0-clx-intel/lib -D TPL_ENABLE_MUMPS:BOOL=0 -D MUMPS_I
NCLUDE_DIRS= -D MUMPS_LIBRARY_DIRS= -D MUMPS_LIBRARY_NAMES:STRING= -D VLE_TPL_MUMPS_LIBRARIES:STRING= -D TPL_ParMETIS_LIBRARIES= -D TPL_ParMETIS_INCLUDE_DIRS= -D TPL_ENABLE_yaml-cpp:BOOL=0 -D ya
ml-cpp_INCLUDE_DIRS:PATH= -D yaml-cpp_LIBRARY_DIRS:PATH= -D Trilinos_ENABLE_Amesos:BOOL=ON -D Trilinos_ENABLE_Amesos2:BOOL=ON -D Amesos2_ENABLE_KLU2:BOOL=ON -D Amesos2_ENABLE_Basker:BOOL=ON -D T
rilinos_ENABLE_Anasazi:BOOL=ON -D Trilinos_ENABLE_AztecOO:Bool=ON -D Trilinos_ENABLE_Belos:BOOL=ON -D Trilinos_ENABLE_Epetra:Bool=ON -D Trilinos_ENABLE_EpetraExt:Bool=ON -D Epetra_ENABLE_TESTS:B
OOL=ON -D Trilinos_ENABLE_FEI:Bool=ON -D Trilinos_ENABLE_Ifpack:Bool=ON -D Trilinos_ENABLE_Ifpack2:BOOL=ON -D Trilinos_ENABLE_Intrepid:BOOL=ON -D Trilinos_ENABLE_Intrepid2:BOOL=ON -D Intrepid_EN
ABLE_TESTS:BOOL=ON -D Trilinos_ENABLE_Isorropia:BOOL=ON -D Trilinos_ENABLE_ML:BOOL=ON -D ML_TAKES_SUPERLU_LESS_THAN_5=TRUE -D ML_ENABLE_SuperLU:BOOL=OFF -D Trilinos_ENABLE_MOOCHO:BOOL=ON -D Tril
inos_ENABLE_MueLu:BOOL= -D MueLu_ENABLE_Tutorial:BOOL=OFF -D MueLu_ENABLE_EXAMPLES:BOOL=OFF -D Trilinos_ENABLE_NOX=ON -D NOX_ENABLE_TESTS:BOOL=OFF -D Trilinos_ENABLE_Pamgen:Bool=ON -D Trilinos_ENABLE_Panzer:Bool=ON -D Trilinos_ENABLE_Phalanx:BOOL=ON -D Phalanx_EXPLICIT_TEMPLATE_INSTANTIATION=ON -D Phalanx_ENABLE_EXAMPLES=OFF -D Trilinos_ENABLE_Piro:BOOL=ON -D Trilinos_ENABLE_Rythmos:BOOL=ON -D Trilinos_ENABLE_Sacado:Bool=ON -D Trilinos_ENABLE_SEACAS:BOOL=0 -D Trilinos_ENABLE_SEACASIoss:BOOL=0 -D Trilinos_ENABLE_SEACASBlot:BOOL=0 -D Trilinos_ENABLE_SEACASExodus:BOOL=0 -D Trilinos_ENABLE_SECONDARY_STABLE_CODE:BOOL=ON -D Trilinos_ENABLE_Shards:BOOL=ON -D Trilinos_ENABLE_ShyLU:BOOL=OFF -D Trilinos_ENABLE_Stokhos:BOOL=ON -D Trilinos_ENABLE_Stratimikos:BOOL=ON -D Trilinos_ENABLE_Teko:BOOL=ON -D Trilinos_ENABLE_Teuchos:BOOL=ON -D Teuchos_ENABLE_LONG_LONG_INT:BOOL=ON -D Trilinos_ENABLE_Thyra:BOOL=ON -D Trilinos_ENABLE_Tpetra:BOOL=ON -D Trilinos_ENABLE_TriKota:BOOL=ON -D Trilinos_ENABLE_Zoltan:BOOL=ON -D Trilinos_ENABLE_Zoltan2:BOOL=ON -D Trilinos_ENABLE_Kokkos:BOOL=ON -D Trilinos_ENABLE_KokkosCore:BOOL=ON -D Phalanx_KOKKOS_DEVICE_TYPE:STRING=SERIAL -D Phalanx_INDEX_SIZE_TYPE:STRING=INT -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF -D Kokkos_ENABLE_Serial:BOOL=ON -D Kokkos_ENABLE_OpenMP:BOOL=ON -D Kokkos_ENABLE_Pthread:BOOL=OFF -D Trilinos_ENABLE_STK:BOOL=0 -D Trilinos_ENABLE_STKIO:BOOL=0 -D Trilinos_ENABLE_STKMesh:BOOL=0 -D Trilinos_ENABLE_PyTrilinos:Bool=ON -D PYTHON_EXECUTABLE:PATH=/scratch1/projects/compilers/oneapi_2021.4.0.3422/intelpython/python3.7/bin/python3 -D CMAKE_PYTHON_INCLUDE_DIR:PATH=/scratch1/projects/compilers/oneapi_2021.4.0.3422/intelpython/python3.7/include -D CMAKE_PYTHON_LIBRARIES:STRING=/scratch1/projects/compilers/oneapi_2021.4.0.3422/intelpython/python3.7/lib -D PyTrilinos_DOCSTRINGS:BOOL=OFF -D PyTrilinos_ENABLE_Tpetra:BOOL=OFF -D SWIG_EXECUTABLE:FILEPATH=/work2/00434/eijkhout/swig/installation-4.1.1-clx-intel/bin/swig -D Trilinos_EXTRA_LD_FLAGS= /work2/00434/eijkhout/trilinos/trilinos-13.4.1

[cgcgcg]

@trilinos/panzer @rppawlo

[rppawlo]

@VictorEijkhout - Thanks for submitting this issue. I'm not sure of the exact problem yet, but suspect that if you change the configure variable: -D Phalanx_KOKKOS_DEVICE_TYPE:STRING=SERIAL to: -D Phalanx_KOKKOS_DEVICE_TYPE:STRING=OPENMP that you may be able to get past this issue. If you get a chance to test this an can give feedback, it would be a big help.

[VictorEijkhout]

Good guess. It made it to 100% and then there was a problem

[100%] Generating Teuchos.RCP.i

so I'll try without pytrilinos. That's another ticket I have.

In case it matters this is the Intel classic oneapi_2022.3.0 compiler.

[VictorEijkhout]

Without python the whole thing goes through. Thanks!

[rppawlo]

Glad it is working! The issue with Phalanx_KOKKOS_DEVICE_TYPE is that panzer must be using the default kokkos execution space somewhere while phalanx wants the execution space specified by configure flag. This might be a quick fix or it might take some time to make everything consistent. For now, another way to fix this is to just not specify the phalanx flag. In this case, the execution space should always be consistent between packages since they both fall back on the default kokkos execution space.

[github-actions[bot]]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE. If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[github-actions[bot]]

This issue was closed due to inactivity for 395 days.