search

GEOS-DEV / LvArray

Portable HPC Containers (C++)
BSD 3-Clause "New" or "Revised" License
48 stars 10 forks source link

Crash during sparsity pattern creation #321

Open victorapm opened 5 months ago

victorapm commented 5 months ago

This issue arises for compositionalMultiphaseFlow at large scale, e.g., level6 problem

Error message on Frontier:

------------------- TIMESTEP START -------------------
    - Time:       00h00m00s (0 s)
    - Delta Time: 00h00m01s (1 s)
    - Cycle:      0
------------------------------------------------------

Time: 0.00e+00 s, dt: 1 s, Cycle: 0

compflowStatistics (time 1 s): Max phase CFL number: 0.0003955364110866208
compflowStatistics (time 1 s): Max component CFL number: 0.00039553641108662084
free(): corrupted unsorted chunks
free(): corrupted unsorted chunks
free(): corrupted unsorted chunks
MPICH ERROR [Rank 1726] [job id 2002214.0] [Wed Jun  5 23:28:00 2024] [frontier00918] - Abort(1) (rank 1726 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1726

aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1726
Received signal 11: Segmentation fault

** StackTrace of 10 frames **
Frame 0: /lib64/libpthread.so.0 
Frame 1: geos::DofManager::setSparsityPattern(LvArray::SparsityPattern<long long, int, LvArray::ChaiBuffer>&) const 
Frame 2: geos::SolverBase::setupSystem(geos::DomainPartition&, geos::DofManager&, LvArray::CRSMatrix<double, long long, int, LvArray::ChaiBuffer>&, geos::HypreVector&, geos::HypreVector&, bool) 
Frame 3: geos::SolverBase::solverStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 4: geos::SolverBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 5: geos::EventBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 6: geos::EventManager::run(geos::DomainPartition&) 
Frame 7: geos::GeosxState::run() 
Frame 8: main 
Frame 9: __libc_start_main 
Frame 10: _start 
=====

Inspecting the core file with gdb:

Thread 1 "geosx" received signal SIGSEGV, Segmentation fault.
0x00007fffec0fc424 in LvArray::arrayManipulation::emplace<long long, long long const&> (ptr=0x0, size=<optimized out>, index=0, args=@0x7fffffff0798: 10356060)
    at /lustre/orion/geo143/scratch/victorapm/projects/geos-dev/src/coreComponents/LvArray/src/arrayManipulation.hpp:489
489   new ( ptr + index ) T( std::forward< ARGS >( args ) ... );

Thread 1 "geosx" received signal SIGSEGV, Segmentation fault.
0x00007fffec0fc424 in LvArray::arrayManipulation::emplace<long long, long long const&> (ptr=0x0, size=<optimized out>, index=0, args=@0x7fffffff0798: 0)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/LvArray/src/arrayManipulation.hpp:489
489   new ( ptr + index ) T( std::forward< ARGS >( args ) ... );
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.31-150300.68.1.x86_64 krb5-debuginfo-1.19.2-150400.3.9.1.x86_64 libatomic1-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libbrotlicommon1-debuginfo-1.0.7-3.3.1.x86_64 libbrotlidec1-debuginfo-1.0.7-3.3.1.x86_64 libcom_err2-debuginfo-1.46.4-150400.3.3.1.x86_64 libcurl4-debuginfo-8.0.1-150400.5.41.1.x86_64 libelf1-debuginfo-0.185-150400.5.3.1.x86_64 libgcc_s1-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libgfortran5-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libidn2-0-debuginfo-2.2.0-3.6.1.x86_64 libjitterentropy3-debuginfo-3.4.0-150000.1.9.1.x86_64 libjson-c3-debuginfo-0.13-3.3.1.x86_64 libkeyutils1-debuginfo-1.6.3-5.6.1.x86_64 libldap-2_4-2-debuginfo-2.4.46-150200.14.17.1.x86_64 libncurses6-debuginfo-6.1-150000.5.20.1.x86_64 libnghttp2-14-debuginfo-1.40.0-150200.17.1.x86_64 libnl3-200-debuginfo-3.3.0-1.29.x86_64 libnuma1-debuginfo-2.0.14.20.g4ee5e0c-150400.1.24.x86_64 libopenssl1_1-debuginfo-1.1.1l-150400.7.63.1.x86_64 libpcre1-debuginfo-8.45-150000.20.13.1.x86_64 libpsl5-debuginfo-0.20.1-150000.3.3.1.x86_64 libselinux1-debuginfo-3.1-150400.1.69.x86_64 libstdc++6-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libyaml-0-2-debuginfo-0.1.7-1.17.x86_64 libz1-debuginfo-1.2.11-150000.3.48.1.x86_64 libzstd1-debuginfo-1.5.0-150400.3.3.1.x86_64
Missing separate debuginfos, use: zypper install glibc-debuginfo-2.31-150300.68.1.x86_64 krb5-debuginfo-1.19.2-150400.3.9.1.x86_64 libatomic1-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libbrotlicommon1-debuginfo-1.0.7-3.3.1.x86_64 libbrotlidec1-debuginfo-1.0.7-3.3.1.x86_64 libcom_err2-debuginfo-1.46.4-150400.3.3.1.x86_64 libcurl4-debuginfo-8.0.1-150400.5.41.1.x86_64 libelf1-debuginfo-0.185-150400.5.3.1.x86_64 libgcc_s1-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libgfortran5-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libidn2-0-debuginfo-2.2.0-3.6.1.x86_64 libjitterentropy3-debuginfo-3.4.0-150000.1.9.1.x86_64 libjson-c3-debuginfo-0.13-3.3.1.x86_64 libkeyutils1-debuginfo-1.6.3-5.6.1.x86_64 libldap-2_4-2-debuginfo-2.4.46-150200.14.17.1.x86_64 libncurses6-debuginfo-6.1-150000.5.20.1.x86_64 libnghttp2-14-debuginfo-1.40.0-150200.17.1.x86_64 libnl3-200-debuginfo-3.3.0-1.29.x86_64 libnuma1-debuginfo-2.0.14.20.g4ee5e0c-150400.1.24.x86_64 libopenssl1_1-debuginfo-1.1.1l-150400.7.63.1.x86_64 libpcre1-debuginfo-8.45-150000.20.13.1.x86_64 libpsl5-debuginfo-0.20.1-150000.3.3.1.x86_64 libselinux1-debuginfo-3.1-150400.1.69.x86_64 libstdc++6-debuginfo-13.2.1+git8285-150000.1.9.1.x86_64 libyaml-0-2-debuginfo-0.1.7-1.17.x86_64 libz1-debuginfo-1.2.11-150000.3.48.1.x86_64 libzstd1-debuginfo-1.5.0-150400.3.3.1.x86_64
bt
(gdb)
 #0  0x00007fffec0fc424 in LvArray::arrayManipulation::emplace<long long, long long const&> (ptr=0x0, size=<optimized out>, index=0, args=@0x7fffffff0798: 0)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/LvArray/src/arrayManipulation.hpp:489
#1  LvArray::sortedArrayManipulation::insert<long long, LvArray::SparsityPattern<long long, int, LvArray::ChaiBuffer>::CallBacks> (ptr=<optimized out>,
    size=<optimized out>, value=@0x7fffffff0798: 0, callBacks=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/LvArray/src/sortedArrayManipulation.hpp:653
#2  0x00007fffecc085c8 in LvArray::ArrayOfSetsView<long long, int, LvArray::ChaiBuffer>::insertIntoSetImpl<LvArray::SparsityPattern<long long, int, LvArray::ChaiBuffer>::CallBacks> (this=0x7fffffff2850, i=0, value=@0x7fffffff0798: 0,
    cbacks=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/LvArray/src/ArrayOfSetsView.hpp:371
#3  LvArray::SparsityPattern<long long, int, LvArray::ChaiBuffer>::insertNonZero (this=0x7fffffff2850, row=0, col=0)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/LvArray/src/SparsityPattern.hpp:327
#4  geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::complete (this=0x7fffffff1c58, stack=..., k=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/SparsityKernelBase.hpp:126
#5  geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}::operator()(int) const (
    this=this@entry=0x7fffffff1c58, k=k@entry=0)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/SparsityKernelBase.hpp:165
#6  0x00007fffecc07f3a in RAJA::policy::sequential::forall_impl<RAJA::TypedRangeSegment<int, int>, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}, camp::resources::v1::Host, RAJA::expt::ForallParamPack<> >(camp::resources::v1::Host, RAJA::policy::sequential::seq_exec const&, RAJA::TypedRangeSegment<int, int>&&, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}&&, RAJA::expt::ForallParamPack<>) (body=..., res=..., iter=...)
    at /ccs/home/victorapm/projects/spack/opt/spack/install-sles15-zen3-cce-17.0.0-rocm_5.4.3/raja-2024.02.0/include/RAJA/policy/sequential/forall.hpp:99
#7  RAJA::wrap::forall<camp::resources::v1::Host, RAJA::policy::sequential::seq_exec, RAJA::TypedRangeSegment<int, int>, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}, RAJA::expt::ForallParamPack<>&>(camp::resources::v1::Host, RAJA::policy::sequential::seq_exec&&, RAJA::TypedRangeSegment<int, int>&&, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}&&, RAJA::expt::ForallParamPack<>&) (loop_body=..., r=..., p=..., c=..., f_params=...)
    at /ccs/home/victorapm/projects/spack/opt/spack/install-sles15-zen3-cce-17.0.0-rocm_5.4.3/raja-2024.02.0/include/RAJA/pattern/forall.hpp:163
#8  RAJA::policy_by_value_interface::forall<RAJA::policy::sequential::seq_exec, camp::resources::v1::Host, RAJA::TypedRangeSegment<int, int>, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}>(RAJA::policy::sequential::seq_exec&&, camp::resources::v1::Host, RAJA::TypedRangeSegment<int, int>&&, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}&&) (params=..., p=..., r=..., c=...)
    at /ccs/home/victorapm/projects/spack/opt/spack/install-sles15-zen3-cce-17.0.0-rocm_5.4.3/raja-2024.02.0/include/RAJA/pattern/forall.hpp:537
#9  RAJA::forall<RAJA::policy::sequential::seq_exec, RAJA::TypedRangeSegment<int, int>, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}, camp::resources::v1::Host>(RAJA::TypedRangeSegment<int, int>&&, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}&&) (args=..., args=...)
    at /ccs/home/victorapm/projects/spack/opt/spack/install-sles15-zen3-cce-17.0.0-rocm_5.4.3/raja-2024.02.0/include/RAJA/pattern/forall.hpp:578
#10 geos::forAll<RAJA::policy::sequential::seq_exec, int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}>(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> >(int, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> const&)::{lambda(int)#1}&&) (end=<optimized out>, body=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/common/GEOS_RAJA_Interface.hpp:223
#11 geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3>::kernelLaunch<RAJA::policy::sequential::seq_exec, geos::finiteElement::SparsityKernelBase<geos::CellElementSubRegion, geos::constitutive::NullModel, geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, 3, 3> > (numElems=numElems@entry=100032, kernelComponent=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/SparsityKernelBase.hpp:157
#12 0x00007fffecba2b46 in geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, geos::CellElementSubRegion&) const::{lambda(auto:1&)#1}::operator()<geos::constitutive::NullModel>(geos::CellElementSubRegion&) const::{lambda(auto:1)#1}::operator()<geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2>(geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2) const (finiteElement=..., this=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/KernelBase.hpp:454
#13 geos::finiteElement::FiniteElementDispatchHandler<geos::finiteElement::H1_Hexahedron_Lagrange1_GaussLegendre2, geos::finiteElement::H1_Wedge_Lagrange1_Gauss6, geos::finiteElement::H1_Tetrahedron_Lagrange1_Gauss1, geos::finiteElement::H1_Pyramid_Lagrange1_Gauss5, geos::finiteElement::ConformingVirtualElementOrder1<4, 3> >::dispatch3D<geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, geos::CellElementSubRegion&) const::{lambda(auto:1&)#1}::operator()<geos::constitutive::NullModel>(geos::CellElementSubRegion&) const::{lambda(auto:1)#1}>(geos::finiteElement::FiniteElementBase&, geos::CellElementSubRegion&&)
    (input=..., lambda=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/FiniteElementDispatch.hpp:168
#14 geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, geos::CellElementSubRegion&) const::{lambda(auto:1&)#1}::operator()<geos::constitutive::NullModel>(geos::CellElementSubRegion&) const (
    castedConstitutiveRelation=..., this=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/KernelBase.hpp:430
#15 geos::constitutive::ConstitutivePassThru<geos::constitutive::NullModel>::execute<geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, geos::CellElementSubRegion&) const::{lambda(auto:1&)#1}>(geos::constitutive::ConstitutiveBase&, geos::CellElementSubRegion&&) (constitutiveRelation=..., lambda=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/constitutive/ConstitutivePassThru.hpp:206
#16 geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, geos::CellElementSubRegion&) const (targetRegionIndex=0, elementSubRegion=..., this=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/KernelBase.hpp:415
#17 geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}::operator()<geos::CellElementSubRegion>(int, int, int, geos::ElementRegionBase&, geos::CellElementSubRegion&) const (targetIndex=0, subRegion=..., this=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:567
#18 geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}::operator()(int, geos::ElementRegionBase&) const::{lambda(auto:1&)#1}::operator()<geos::CellElementSubRegion>(geos::CellElementSubRegion&) const (castedSubRegion=..., this=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:718
#19 geos::dataRepository::Group::applyLambdaToContainer<geos::CellElementSubRegion, geos::ElementSubRegionBase, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}::operator()(int, geos::ElementRegionBase&) const::{lambda(auto:1&)#1}>(geos::ElementSubRegionBase&, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}::operator()(int, geos::ElementRegionBase&) const::{lambda(auto:1&)#1}&&) (container=..., lambda=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/dataRepository/Group.hpp:452
#20 geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}::operator()(int, geos::ElementRegionBase&) const (targetIndex=0, elementRegion=..., this=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:716
#21 geos::dataRepository::Group::forSubGroups<geos::ElementRegionBase, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(auto:1&)#1}::operator()<geos::ElementRegionBase>(geos::ElementRegionBase&) const (castedSubGroup=..., this=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/dataRepository/Group.hpp:594
#22 geos::dataRepository::Group::applyLambdaToContainer<geos::ElementRegionBase, geos::dataRepository::Group, geos::dataRepository::Group::forSubGroups<geos::ElementRegionBase, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(auto:1&)#1}>(geos::dataRepository::Group&, geos::dataRepository::Group::forSubGroups<geos::ElementRegionBase, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKer#20 geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}::operator()(int, geos::ElementRegionBase&) const (targetIndex=0, elementRegion=..., this=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:716
#23 geos::dataRepository::Group::forSubGroups<geos::ElementRegionBase, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&) (this=0x7c08560, subGroupKeys=..., lambda=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/dataRepository/Group.hpp:592
#24 geos::ElementRegionManager::forElementRegions<geos::ElementRegionBase, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&) (
    this=<optimized out>, targetRegions=..., lambda=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:312
#25 geos::ElementRegionManager::forElementSubRegionsComplete<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&) (this=<optimized out>, targetRegions=...,
    lambda=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:706
#26 geos::ElementRegionManager::forElementSubRegions<geos::CellElementSubRegion, , LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer>, geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> >(geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&)::{lambda(int, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&) (this=<optimized out>,
    targetRegions=..., lambda=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mesh/ElementRegionManager.hpp:560
nels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, int, int, geos::ElementRegionBase&, auto:1&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(int, geos::ElementRegionBase&)#1}>(LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic>&&)::{lambda(auto:1&)#1}&&)
    (container=..., lambda=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/dataRepository/Group.hpp:452
#27 geos::finiteElement::regionBasedKernelApplication<RAJA::policy::sequential::seq_exec, geos::constitutive::NullModel, geos::CellElementSubRegion, geos::finiteElement::SparsityKernelFactory<geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> > (mesh=..., targetRegions=...,
    finiteElementName=..., constitutiveStringName=..., kernelFactory=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/KernelBase.hpp:387
#28 geos::finiteElement::fillSparsity<geos::CellElementSubRegion, geos::solidMechanicsLagrangianFEMKernels::ImplicitSmallStrainQuasiStatic> (mesh=...,
    targetRegions=..., discretizationName=..., inputDofNumber=...,
    rankOffset=10362690, inputSparsityPattern=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/finiteElement/kernelInterface/SparsityKernelBase.hpp:296
#29 geos::SolidMechanicsLagrangianFEM::setupSystem(geos::DomainPartition&, geos::DofManager&, LvArray::CRSMatrix<double, long long, int, LvArray::ChaiBuffer>&, geos::HypreVector&, geos::HypreVector&, bool)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&)#1}::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&) const (this=0x7fffffff2830, mesh=..., regionNames=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/physicsSolvers/solidMechanics/SolidMechanicsLagrangianFEM.cpp:997
#30 geos::SolverBase::forDiscretizationOnMeshTargets<geos::SolidMechanicsLagrangianFEM::setupSystem(geos::DomainPartition&, geos::DofManager&, LvArray::CRSMatrix<double, long long, int, LvArray::ChaiBuffer>&, geos::HypreVector&, geos::HypreVector&, bool)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&)#1}>(geos::dataRepository::Group&, geos::SolidMechanicsLagrangianFEM::setupSystem(geos::DomainPartition&, geos::DofManager&, LvArray::CRSMatrix<double, long long, int, LvArray::ChaiBuffer>&, geos::HypreVector&, geos::HypreVector&, bool)::{lambda(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, geos::MeshLevel&, LvArray::ArrayView<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, 1, 0, int, LvArray::ChaiBuffer> const&)#1}&&) const (
    this=this@entry=0x7c5d500, meshBodies=..., lambda=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/physicsSolvers/SolverBase.hpp:723
#31 0x00007fffecba09ba in geos::SolidMechanicsLagrangianFEM::setupSystem (
    this=0x7c5d500, domain=..., dofManager=..., localMatrix=..., rhs=...,
    solution=..., setSparsity=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/physicsSolvers/solidMechanics/SolidMechanicsLagrangianFEM.cpp:970
#32 0x00007fffecb98548 in geos::SolidMechanicsLagrangianFEM::solverStep (
    this=0x7c5d500, time_n=@0x7fffffff2c20: 0, dt=@0x7fffffff2b50: 1,
    cycleNumber=0, domain=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/physicsSolvers/solidMechanics/SolidMechanicsLagrangianFEM.cpp:486
#33 0x00007fffec2fd6f5 in geos::SolverBase::execute (this=0x7c5d500,
    time_n=<error reading variable: That operation is not available on integers of more than 8 bytes.>,
    dt=<error reading variable: That operation is not available on integers of more than 8 bytes.>, cycleNumber=0, domain=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/physicsSolvers/SolverBase.cpp:252
#34 0x00007fffed8a76dc in geos::EventBase::execute (this=0x7c56cd0,
    time_n=<error reading variable: That operation is not available on integers of more than 8 bytes.>,
    dt=<error reading variable: That operation is not available on integers of more than 8 bytes.>, cycleNumber=0, domain=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/events/EventBase.cpp:235
#35 0x00007fffed8abd67 in geos::EventManager::run (this=0x7cba4c0, domain=...)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/events/EventManager.cpp:195
#36 0x00007fffed8bb9d5 in geos::GeosxState::run (this=0x7fffffff3430)
    at PATH_TO_GEOS/projects/geos-dev/src/coreComponents/mainInterface/GeosxState.cpp:179
#37 0x000000000020e724 in main (argc=5, argv=<optimized out>)
    at PATH_TO_GEOS/projects/geos-dev/src/main/main.cpp:50
corbett5 commented 5 months ago

Just looking at the first few lines of the gdb output it looks like arrayManipulation::emplace is being passed a nullptr. Unfortunately the few frames above this argument is optimized out so you can't tell where it is introduced. Any chance this problem is running out of memory, I imagine malloc returning a nullptr might cause this exact problem.

victorapm commented 5 months ago

Thanks, Ben!

It doesn't seem this problem runs out of memory since the high watermarks for the level5 version of it (coarser by a factor of 2 in each direction) look good:

Umpire          DEVICE sum across ranks: 22211.7 G
Umpire          DEVICE         rank max:   11.4 GB
Umpire       DEVICE::0 sum across ranks: 22211.7 G
Umpire       DEVICE::0         rank max:   11.4 GB
Umpire            HOST sum across ranks: 13830.5 G
Umpire            HOST         rank max:    6.9 GB
Umpire    HYPRE_DEVICE sum across ranks: 12683.7 G
Umpire    HYPRE_DEVICE         rank max:    6.5 GB
Umpire          PINNED sum across ranks: 1173.8 GB
Umpire          PINNED         rank max:  671.9 MB

i.e., we have 64 GB available on the GPU and use only about 18 GB. But, anyway, it would probably be good to double-check that...

I guess recompiling in full Debug mode and perhaps with -DLVARRAY_BOUNDS_CHECK will help us with more info

corbett5 commented 5 months ago

Yeah after a tad more thought I doubt it's an OOM error since in those cases we usually see the error from Umpire directly. But certainly if it's not terribly slow running in full debug mode will hopefully tell us more.

rrsettgast commented 5 months ago

I think Debug mode will be horrendously slow for this case. Perhaps some more targeted debuting via printf statements are in order here? Maybe start with the insertNonZero and everything "finer" that. It will be a bit of a mess, but you stand a better chance than with a debug run at 1/4 frontier scale.