search

idaholab / moose

Multiphysics Object Oriented Simulation Environment
https://www.mooseframework.org
GNU Lesser General Public License v2.1
1.78k stars 1.05k forks source link

GrainTracker + Distributed Mesh + Periodic BCs == failure #10380

Open permcody opened 6 years ago

permcody commented 6 years ago

Rationale

The GrainTracker doesn't work when using DistributedMesh if the simulation also contains Periodic BCs. The issue comes from the fact that the GrainTracker never expects to deal with remote elements (it's been designed without the need for them). However, in the topological case, those pointers are defined as RemoteElems which are not NULL, but also can't be used for a lot of different methods.

Description

Add mesh/parallel_type = distributed to grain_tracker_test_elemental.i. This will trigger an assertion in debug/devel mode and will likely crash in optimized mode.

This also affects the halo visualizations: compute_halo_maps = true for the same reasons. The issue has to do with calculating the halos across periodic boundaries (https://github.com/idaholab/moose/pull/8926) which is a feature that was added and is necessary on larger problems. I'm calling methods on remote elems that I shouldn't be. I'm not sure how to fix this just yet.

Impact

Large 3D simulations will not be able to use periodic boundary conditions until this design is patched up.

SudiptaBiswas commented 5 years ago

I recently faced the same issue while trying to use periodic BC for large 3D grain growth simulation. I was using pre-split mesh. The simulation seems to hang at the initial problem setup. Here is the stack,

#0  0x00002aaab0793c00 in psm_progress_wait () from /apps/local/easybuild/software/MVAPICH2/2.3-GCC-5.4.0-2.28/lib/libmpi.so.12
#1  0x00002aaab071e6c5 in MPIC_Wait () from /apps/local/easybuild/software/MVAPICH2/2.3-GCC-5.4.0-2.28/lib/libmpi.so.12
#2  0x00002aaab071ee51 in MPIC_Sendrecv () from /apps/local/easybuild/software/MVAPICH2/2.3-GCC-5.4.0-2.28/lib/libmpi.so.12
#3  0x00002aaab05b3fa1 in MPIR_Allgatherv_Bruck_MV2 () from /apps/local/easybuild/software/MVAPICH2/2.3-GCC-5.4.0-2.28/lib/libmpi.so.12
#4  0x00002aaab05b58a5 in MPIR_Allgatherv_MV2 () from /apps/local/easybuild/software/MVAPICH2/2.3-GCC-5.4.0-2.28/lib/libmpi.so.12
#5  0x00002aaab0578d80 in MPIR_Allgatherv_impl () from /apps/local/easybuild/software/MVAPICH2/2.3-GCC-5.4.0-2.28/lib/libmpi.so.12
#6  0x00002aaab05796a9 in PMPI_Allgatherv () from /apps/local/easybuild/software/MVAPICH2/2.3-GCC-5.4.0-2.28/lib/libmpi.so.12
#7  0x00002aaaac897c94 in void libMesh::Parallel::Communicator::allgather<unsigned long, std::allocator<unsigned long> >(std::vector<unsigned long, std::allocator<unsigned long> >&, bool) const () from /home/bisws/projects/lemhi/moose/framework/libmoose-opt.so.0
#8  0x00002aaaace5a92d in MooseMesh::ghostGhostedBoundaries() () from /home/bisws/projects/lemhi/moose/framework/libmoose-opt.so.0
#9  0x00002aaaacc2de70 in FEProblemBase::ghostGhostedBoundaries() () from /home/bisws/projects/lemhi/moose/framework/libmoose-opt.so.0
#10 0x00002aaaacc6115a in FEProblemBase::init() () from /home/bisws/projects/lemhi/moose/framework/libmoose-opt.so.0
#11 0x00002aaaacb29f64 in Action::timedAct() () from /home/bisws/projects/lemhi/moose/framework/libmoose-opt.so.0
#12 0x00002aaaacb2edd5 in ActionWarehouse::executeActionsWithAction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /home/bisws/projects/lemhi/moose/framework/libmoose-opt.so.0
#13 0x00002aaaacb300ec in ActionWarehouse::executeAllActions() () from /home/bisws/projects/lemhi/moose/framework/libmoose-opt.so.0
#14 0x00002aaaad1f83db in MooseApp::runInputFile() () from /home/bisws/projects/lemhi/moose/framework/libmoose-opt.so.0
#15 0x00002aaaad1f3f80 in MooseApp::run() () from /home/bisws/projects/lemhi/moose/framework/libmoose-opt.so.0
#16 0x0000000000402657 in main ()