Failing ACE MiniErosion Denudation test in parallel

[ikalash]

The ACE_MiniErosion_Denudation_Parallel is failing with the following cryptic STK-related error:

3: STKDisc: sideset bottom has size 0  on Proc 0.
3: 
3: p=0: *** Caught standard std::exception of type 'std::logic_error' :
3: 
3:  Requirement( buf.remaining() ) FAILED
3:  Error occurred at: stk_mesh/stk_mesh/baseImpl/MeshCommImplUtils.cpp:215
3:   0# stk::output_stacktrace(std::ostream&) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_util_util.so.13
3:   1# stk::mesh::impl::unpack_induced_parts_from_sharers(std::vector<unsigned int, std::allocator<unsigned int> >&, std::vector<stk::mesh::EntityCommInfo, std::allocator<stk::mesh::EntityCommInfo> > const&, stk::CommSparse&, stk::mesh::EntityKey) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   2# stk::mesh::BulkData::remove_unneeded_induced_parts(stk::mesh::Entity, std::vector<stk::mesh::EntityCommInfo, std::allocator<stk::mesh::EntityCommInfo> > const&, stk::mesh::PartStorage&, stk::CommSparse&) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   3# stk::mesh::BulkData::internal_resolve_shared_part_membership_for_element_death() in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   4# stk::mesh::BulkData::internal_resolve_sharing_and_ghosting_for_sides(bool) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   5# stk::mesh::BulkData::make_mesh_parallel_consistent_after_skinning(std::vector<stk::mesh::sharing_info, std::allocator<stk::mesh::sharing_info> > const&) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   6# stk::mesh::FaceCreator::create_side_entities_given_sideset(stk::mesh::SideSet const&, std::vector<stk::mesh::Part*, std::allocator<stk::mesh::Part*> > const&) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   7# stk::mesh::create_all_sides(stk::mesh::BulkData&, stk::mesh::Selector const&, std::vector<stk::mesh::Part*, std::allocator<stk::mesh::Part*> > const&, bool) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   8# stk::mesh::create_adjacent_entities(stk::mesh::BulkData&, std::vector<stk::mesh::Part*, std::allocator<stk::mesh::Part*> >&) in /home/ikalash/Trilinos-clean/build/install/lib/libstk_mesh_base.so.13
3:   9# LCM::Topology::createAllLevelsRelations() in /home/ikalash/Albany-LCM/build/src/LCM/libLCM.so
3:  10# LCM::Topology::graphInitialization() in /home/ikalash/Albany-LCM/build/src/LCM/libLCM.so
3:  11# LCM::Topology::Topology(Teuchos::RCP<Albany::AbstractDiscretization>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in /home/ikalash/Albany-LCM/build/src/LCM/libLCM.so
3:  12# AAdapt::Erosion::Erosion(Teuchos::RCP<Teuchos::ParameterList> const&, Teuchos::RCP<Sacado::ScalarParameterLibrary<SPL_Traits> > const&, Albany::StateManager const&, Teuchos::RCP<Teuchos::Comm<int> const> const&) in /home/ikalash/Albany-LCM/build/src/adapt/libalbanyAdapt.so
3:  13# AAdapt::AdaptiveSolutionManager::buildAdapter(Teuchos::RCP<AAdapt::rc::Manager> const&) in /home/ikalash/Albany-LCM/build/src/adapt/libalbanyAdapt.so
3:  14# AAdapt::AdaptiveSolutionManager::AdaptiveSolutionManager(Teuchos::RCP<Teuchos::ParameterList> const&, Teuchos::RCP<Thyra::VectorBase<double> const> const&, Teuchos::RCP<Sacado::ScalarParameterLibrary<SPL_Traits> > const&, Albany::StateManager const&, Teuchos::RCP<AAdapt::rc::Manager> const&, Teuchos::RCP<Teuchos::Comm<int> const> const&) in /home/ikalash/Albany-LCM/build/src/adapt/libalbanyAdapt.so
3:  15# Albany::Application::finalSetUp(Teuchos::RCP<Teuchos::ParameterList> const&, Teuchos::RCP<Thyra::VectorBase<double> const> const&) in /home/ikalash/Albany-LCM/build/src/libalbanyLib.so
3:  16# Albany::Application::Application(Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::RCP<Teuchos::ParameterList> const&, Teuchos::RCP<Thyra::VectorBase<double> const> const&, bool) in /home/ikalash/Albany-LCM/build/src/libalbanyLib.so
3:  17# Albany::SolverFactory::createAlbanyAppAndModel(Teuchos::RCP<Albany::Application>&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::RCP<Thyra::VectorBase<double> const> const&, bool) in /home/ikalash/Albany-LCM/build/src/libalbanyLib.so
3:  18# Albany::SolverFactory::createAndGetAlbanyApp(Teuchos::RCP<Albany::Application>&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::RCP<Teuchos::Comm<int> const> const&, Teuchos::RCP<Thyra::VectorBase<double> const> const&, bool) in /home/ikalash/Albany-LCM/build/src/libalbanyLib.so
3:  19# LCM::ACEThermoMechanical::createThermalSolverAppDiscME(int, double) const in /home/ikalash/Albany-LCM/build/src/LCM/libLCM.so
3:  20# LCM::ACEThermoMechanical::ThermoMechanicalLoopDynamics() const in /home/ikalash/Albany-LCM/build/src/LCM/libLCM.so
3:  21# Thyra::ModelEvaluatorDefaultBase<double>::evalModel(Thyra::ModelEvaluatorBase::InArgs<double> const&, Thyra::ModelEvaluatorBase::OutArgs<double> const&) const in /home/ikalash/Albany-LCM/build/src/libalbanyLib.so
3:  22# void Piro::Detail::PerformSolveImpl<double, Thyra::VectorBase<double> const, Thyra::MultiVectorBase<double> const>(Thyra::ModelEvaluator<double> const&, Teuchos::Array<bool> const&, bool, Teuchos::Array<Teuchos::RCP<Thyra::VectorBase<double> const> >&, Teuchos::Array<Teuchos::Array<Teuchos::RCP<Thyra::MultiVectorBase<double> const> > >&, Teuchos::Array<Teuchos::RCP<Thyra::MultiVectorBase<double> const> >&, Teuchos::Array<Teuchos::Array<Teuchos::RCP<Thyra::MultiVectorBase<double> const> > >&, Teuchos::RCP<Piro::SolutionObserverBase<double, Thyra::VectorBase<double> const> >) in /home/ikalash/Albany-LCM/build/src/Albany
3:  23# void Piro::Detail::PerformSolveImpl<double, Thyra::VectorBase<double> const, Thyra::MultiVectorBase<double> const>(Thyra::ModelEvaluator<double> const&, Teuchos::ParameterList&, Teuchos::Array<Teuchos::RCP<Thyra::VectorBase<double> const> >&, Teuchos::Array<Teuchos::Array<Teuchos::RCP<Thyra::MultiVectorBase<double> const> > >&) in /home/ikalash/Albany-LCM/build/src/Albany
3:  24# main in /home/ikalash/Albany-LCM/build/src/Albany
3:  25# __libc_start_main in /lib64/libc.so.6
3:  26# _start in /home/ikalash/Albany-LCM/build/src/Albany
3:  
3:  Error: P0 empty buf, expected to recv parts for: (FACE_RANK,966) from proc 1
3:  

The error happens immediately after the problem starts, and is therefore not an issue related to erosion. The problem does not appear to be related to recent changes in Albany, as the same error is encountered w/o the changes if the .jou file is modified to merge together the mesh, something that was not being done before (leading to incorrect behavior that fell through the cracks).

[ikalash]

@alanw0 : I was wondering if you might have any insight into this issue? Basically we're running a really simple cuboid geometry in Albany, given by the following .jou file: https://github.com/sandialabs/LCM/blob/main/tests/LCM/ACE/MiniErosion/grid/cuboid_denudation.jou . If we add the following 3 lines, which we need for the geometry to be defined correctly and for erosion of the geometry to happen:

https://github.com/sandialabs/LCM/blob/main/tests/LCM/ACE/MiniErosion/grid/cuboid_denudation.jou#L32-L34

the problem does not run in parallel, with the error described in this issue. It happens as soon as the problem starts up, before and computation has been done. Do you have any insight into this from the STK side?

[alanw0]

@ikalash we’ll grab that journal file and take a look, I’ll let you know.

[ikalash]

Thanks @alanw0 , sounds great!

[alanw0]

@ikalash it's taking me a little longer to get to this, we've got a hectic week going on... I'm going to open a sierra ticket to increase the odds that stk-team members can look at this right away.

[ikalash]

No problem - thanks @alanw0 !

[alanw0]

@ikalash how many processors are you using when you see this crash? I took the journal file and made a mesh with cubit, and read the mesh into a simple stk program (it uses stk-io to read into stk-mesh), and called stk::mesh::create_adjacent_entities, and it runs without any trouble on 2, 3, and 4 processors. I have lines 32-34 in the journal file.

[ikalash]

Thanks for checking this @alanw0 . The issue happens for > 1 proc. However, it looks like the problem has to do with the Erosion capabilities, not anything in STK.

@lxmota : if I run this problem thermal-only with Erosion activated, I get the error; if I turn off Erosion, it runs. Also, if I run a similar problem, thermal only through the Thermal 3D problem (which does not have erosion), it runs. Could you please have a look at the issue? I am attaching input files for the three cases I ran, 1 of which fails (thermal_denudation_only.yaml) and 2 of which run (thermal_denudation_only_no_erosion.yaml, input_forward_euler.yaml). input_forward_euler.yaml.txt thermal_denudation_only_no_erosion.yaml.txt thermal_denudation_only.yaml.txt

[alanw0]

ok great, I'll cancel the stk ticket.

[ikalash]

So I've dug into this a bit more using the method of bisection to try to understand when the problem started. What this revealed is that the following line in the .jou file that generates the input mesh file is the one that is causing this problem:

sideset 2 face with (x_coord <= {0 + eps}) and (y_coord > {-L/2 - eps}) and (y_coord < {L/2 + eps}) and (z_coord > {0 - eps}) and (z_coord < {2*L + eps})
sideset 2 name "bluff_face-erodible"

https://github.com/sandialabs/LCM/blob/main/tests/LCM/ACE/MiniErosion/grid/cuboid_denudation.jou#L61-L62

If I replace this with

sideset 2 surface with (x_coord <= {0 + eps})
sideset 2 name "bluff_face-erodible"

then the test case runs. I verified going back to Dec. 2020 that the same holds (with the first two lines, there's an error, with the second two lines, things work fine).

Here is the kicker: in the denudation test where the error happens, there is no BC being set on bluff_face-erodible. We should therefore be able to just remove that from the .jou file and the test should run, which it does. What is weird is that the final solution is not exactly the same as it was when bluff_face_erodible was a SS in the mesh, which does not make any sense to me. @lxmota , any thoughts?

I'm inclined to just remove bluff_face-erodible from the .jou file and rebaseline the test case, but it is very disturbing to me why the presence of that SS would matter and would influence the solution given that no BC is being set on that SS.

[ikalash]

So there is something else that is disturbing. If I remove the definition of bluff_face-erodible in the .jou file and run on 4 procs, then again on 1 proc, the results are different. Attaches is a screenshot - right picture is on 4 procs, left one is on 1 proc. This suggests there is a parallel inconsistency bug, which is disturbing...

We should figure out the dfDT / theta stuff first that seems to be causing the solution to be mesh-dependent, but I wanted to document this parallel issue here for future reference.

[ikalash]

Following up on the discussion at this week's ACE meeting: I have verified that the ACE problems run w/o encountering this error w/o erosion, with the caveat that currently Tempus must be used, as Piro assumes adaptation is happening. I will open a separate issue allowing user to run cases with Piro and no erosion for ACI/NH.

[lxmota]

@ikalash @alanw0 Sorry to come back to this issue after so long. I refer to Irina's findings about the journal file below:

So I've dug into this a bit more using the method of bisection to try to understand when the problem started. What this revealed is that the following line in the .jou file that generates the input mesh file is the one that is causing this problem:

sideset 2 face with (x_coord <= {0 + eps}) and (y_coord > {-L/2 - eps}) and (y_coord < {L/2 + eps}) and (z_coord > {0 - eps}) and (z_coord < {2*L + eps})
sideset 2 name "bluff_face-erodible"

https://github.com/sandialabs/LCM/blob/main/tests/LCM/ACE/MiniErosion/grid/cuboid_denudation.jou#L61-L62

If I replace this with

sideset 2 surface with (x_coord <= {0 + eps})
sideset 2 name "bluff_face-erodible"

then the test case runs. I verified going back to Dec. 2020 that the same holds (with the first two lines, there's an error, with the second two lines, things work fine).

Here is the kicker: in the denudation test where the error happens, there is no BC being set on bluff_face-erodible. We should therefore be able to just remove that from the .jou file and the test should run, which it does. What is weird is that the final solution is not exactly the same as it was when bluff_face_erodible was a SS in the mesh, which does not make any sense to me. @lxmota , any thoughts?

I'm inclined to just remove bluff_face-erodible from the .jou file and rebaseline the test case, but it is very disturbing to me why the presence of that SS would matter and would influence the solution given that no BC is being set on that SS.

The original journal file creates a mesh with a side set that includes all faces, both on the boundary and interior. The side set is created with:

sideset 2 face with (x_coord <= {0 + eps}) and (y_coord > {-L/2 - eps}) and (y_coord < {L/2 + eps}) and (z_coord > {0 - eps}) and (z_coord < {2*L + eps})
sideset 2 name "bluff_face-erodible"

I emphasize that due to the geometry of the body, the limits in that Cubit statement should include every face in the mesh into the side set. This is the result:

This mesh is correct for our purposes because we need a side set with both boundary and interior faces in order to propagate boundary conditions once we start removing elements on the boundary. But this mesh produces the error in STK referenced above. The issue appears to be the call to stk::mesh::create_adjacent_entities in this mesh when it is decomposed.

If instead we create the side set with:

sideset 2 surface with (x_coord <= {0 + eps})
sideset 2 name "bluff_face-erodible"

The mesh has a side set that includes only the faces on the boundary. This seems strange because removing the limits on the Y and Z coordinates should still include all faces of the mesh in the side set due to the geometry of the body. But this is the result:

This mesh is not correct for our purposes because the lack of interior faces on the side set will prevent propagation of boundary conditions when boundary elements are removed. It runs without producing the STK error, but the results are not correct.

So there are two issues:

• A side set specification that should produce the same side set but does not in Cubit.
• A side set with interior faces that produces an error when calling stk::mesh::create_adjacent_entities on a decomposed mesh.

[alanw0]

A couple of comments about this. A newer alternative to create_adjacent_entities is create_all_sides in SkinBoundary.hpp. This might work better, it's worth a try. Another point is that sidesets which include sides that are internal to an element block (i.e., not on the boundary between two element blocks) are technically not supported in stk-mesh. Functions like is_positive_sideset_face_polarity in SidesetUtil.hpp can't work for purely internal sides.

[lxmota]

Ah ok. Thanks for the info. It seems that we need to think of a different way to deal with our BCs other than using internal side sets.