idaholab / moose

Multiphysics Object Oriented Simulation Environment
https://www.mooseframework.org
GNU Lesser General Public License v2.1
1.76k stars 1.05k forks source link

Parallel Mesh By Choice #2105

Closed friedmud closed 10 years ago

friedmud commented 10 years ago

We need to configure libMesh with ParallelMesh always... and then optionally use "parallel = true" in the Mesh block to turn it on. We will default to SerialMesh.

Just don't use Mesh() to create the Mesh... use either SerialMesh or ParallelMesh depending on that option.

jwpeterson commented 10 years ago

In 4e282ef16d4145c5c2a3800b1eac15f19a8b73bd:

Make MooseMesh::_mesh protected and force clients to use the accessor.

Refs #2105.
jwpeterson commented 10 years ago

In 2e96c32fa2876a806b9a0b896367746e122fe653:

Adding documentation in MooseMesh.

Refs #2105.
jwpeterson commented 10 years ago

In 92f695d1280348af743cb8b076de6cd0f627b575:

Move function definitions to C file.

Refs #2105.
jwpeterson commented 10 years ago

In fe82cedaeaa240a83f32c7bbbf3f78a6bc8013e3:

MooseMesh::_mesh is now a pointer, in preparation for letting it
dynamically choose between being a SerialMesh or ParallelMesh.

Refs #2105.
jwpeterson commented 10 years ago

In 8b779427806fbab6d44c6907cad79bd3ce0db814:

Consistently use getMesh().

That way we can insulate code from possible future changes to the type
of MooseMesh::_mesh.

Refs #2105.
jwpeterson commented 10 years ago

In :

Be explicit about the use of SerialMesh where necessary.

Refs #2105.
jwpeterson commented 10 years ago

In b6769946eda5a1d2b60da29474122dd566d9c505:

Don't use LIBMESH_ENABLE_PARMESH to determine whether MooseMesh
holds a ParallelMesh.

MooseMesh will (eventually) be able to contain either a SerialMesh
_or_ a ParallelMesh regardless of the value of LIBMESH_ENABLE_PARMESH,
so we can't rely on that anymore.

Refs #2105.
jwpeterson commented 10 years ago

Since the plan is to use a MooseEnum {PARALLEL, SERIAL, DEFAULT} to read the user's input, we probably don't want to call it 'parallel' any more:

parallel = parallel

or even better

parallel = serial

is just crazy-pants. What do we want to call it? Right now I'm using

mesh_decomposition_type = parallel

But I'm willing to admit that name's a little cumbersome...

YaqiWang commented 10 years ago

or 'distribute_mesh = true/false'

aeslaughter commented 10 years ago

How about something simple: mesh_type

mesh_type = parallel
mesh_type = serial
mesh_type = default
jwpeterson commented 10 years ago

Andrew, I first thought about 'mesh_type' but since we already have the 'type' parameter, I thought it might be too similar? If anyone else thinks mesh_type is fine I'd be happy to use that instead.

Yaqi, your suggestion is good too, but we do need something more than just a boolean type. The 'default' mesh_type means that the input file will work with either SerialMesh or ParallelMesh, and the user can override it by specifying --parallel-mesh on the command line.

andrsd commented 10 years ago

Why not to use just: decomposition = {default | parallel | serial} ?

friedmud commented 10 years ago

I like something close to what Yaqi said:

parallel_distribution = { default | serial | parallel }

or maybe just

distribution = { default | serial | parallel }

"decomposition" is getting too close to "partitioning" I could see people getting confused by that.

jwpeterson commented 10 years ago

OK going with this

distribution = { default | serial | parallel }
jwpeterson commented 10 years ago

In 354392c236522dd64754c53e2b2c8ab4aece50ae:

Picking a better name for _is_parallel (it really means _is_nemesis).

The accessor is removed, but it could always be readded.

'nemesis' is a parameter on MooseMesh instead of FileMesh.

Reformatting long doc strings onto separate lines.

Adding new command line option to MOOSE: --parallel-mesh.

MooseMesh now uses the MooseApp object to determine if
--parallel-mesh has been specified on the command line.

MooseMesh reads an enumeration for the type of mesh "distribution" to use.

Note: if the user does not specify anything for 'distribution' in the
Mesh block, it is possible to set it to parallel via the
--parallel-mesh command line argument, otherwise it defaults to
serial.  This command line arg is not allowed to override the user if
he does specify 'distribution = serial' in the Mesh block.

Refs #2105.
jwpeterson commented 10 years ago

In d9fb1d4ef34346ed1aca9218838be274d2eb5e9f:

Nemesis output can be performed even if the underlying
libMesh mesh is a SerialMesh.

Removing an expect_err test and modifying another so
that it runs for serial meshes as well.

Refs #2105.
jwpeterson commented 10 years ago

In 37fbf05115d2bd309bfa034be4477d87552f6f51:

Transitioning from mesh_mode = 'SERIAL' (which is no longer sufficient)
to adding 'distribution = serial' in Periodic BCs tests.

Periodic BCs do not currently work with ParallelMesh on more than one
processor, but most of these input files will actually work with
ParallelMesh enabled if you run on 1 processor...

Refs #2105.
Refs #2119.
jwpeterson commented 10 years ago

In 4b3f7b12e542994f9f78b4c703fc574c0487ab30:

Transitioning from mesh_mode = 'SERIAL' (which is no longer sufficient)
to adding 'distribution = serial' in all "geomsearch" tests.

The geometric search stuff does not currently work with ParallelMesh on
more than one processor, but most of these input files will actually
work with ParallelMesh enabled if you run on 1 processor...

Refs #2105.
Refs #2121.
jwpeterson commented 10 years ago

In 3c9144287e07a4ed54cb50d948c4a53dade57b22:

ExodusII_IO::write_element_data() is causing a large number of tests
to fail in parallel with ParallelMesh enabled.  If we can fix that in
libmesh, we can hopefully remove the 'distribution = serial' line from
all the input files in this commit.

Refs #2105.
Refs #2122.
jwpeterson commented 10 years ago

In 4da24eea4f0d1ca5dc17166fa5616671edf8759c:

This test no longer needs to be marked with mesh_mode = 'PARALLEL'
since nemesis=true in the Mesh block implies the use of ParallelMesh.

Refs #2105.
jwpeterson commented 10 years ago

In 6e81a40c2faf841dd7acc9d394d557e26b7953ef:

This test has to use SerialMesh because stitch_meshes() is only implemented for SerialMesh.

Note that running:

../../../moose_test-opt -i tiled_mesh_test.i --mesh-only --parallel-mesh

Produces the error:

 Exiting without writing output
 ERROR: Unrecognized file extension: --parallel-mesh

So I think there's something wrong with our command line argument processing...

Refs #2105.
Refs #2124.
jwpeterson commented 10 years ago

In c38f4ef76ce165cb8423d00324bf61ccfbb34400:

The test_parmesh_check is no longer valid, so I removed it from
the chain of prerequisites.

Refs #2105.
jwpeterson commented 10 years ago

In a1f8022dd33ef6b0fdcf5939dca1dbc007dbc1fb:

DiracKernels don't seem to work currently in parallel with ParallelMesh.

The tests in this commit currently segfault when run in parallel with
ParallelMesh, and are therefore restricted to run with SerialMesh.
For details, see #2125.

Refs #2105.
Refs #2125.
jwpeterson commented 10 years ago

In c977ec61d8a79d32c8c3081087d811876876cfdd:

Marking multiapp transfer stuff as serial-only for now.

Refs #2105.
Refs #2126.
jwpeterson commented 10 years ago

In c1a7def30f4c44f2548d8e27a1174a5c4fefed67:

Tests using MeshModifiers are restricted to using SerialMesh.
For more information, see #2129.

Refs #2105.
Refs #2129.
jwpeterson commented 10 years ago

In 01cc5adeda1c42bb24fb3d6360e3e4eb0581dcc6:

Modifying the Gmsh test slightly.

It now only tests reading a mesh in Gmsh format and writing it back
out.  Should also work with ParallelMesh enabled.

Refs #2105.
jwpeterson commented 10 years ago

In a4d80bd21d7cea26ad4d67767a59d0626d76276a:

The mixed-dimension tests are restricted to run with SerialMesh.
For more information, see #2130.

Refs #2105.
Refs #2130.
jwpeterson commented 10 years ago

In 938e89f0fa70eece0f8b838b920d6c5811fe074b:

Make the all_print_pps test run with SerialMesh only.

Apparently just changing the mesh type from SerialMesh to ParallelMesh
is enough to make it do a different number of residual evaluations...

Refs #2105.
jwpeterson commented 10 years ago

In 4e17d46ddef2149c6b5bb175fcfa92dff2fcbc82:

Fixing strange indentation in some test input files.

Refs #2105.
jwpeterson commented 10 years ago

In 9a5aef4c11be07db77221c7fcc4b1c7ea5423daf:

Restricting these tests to run only with SerialMesh for now,
StripeMesh doesn't work if the underlying Mesh is a ParallelMesh.

Refs #2105.
jwpeterson commented 10 years ago

In f92289de7ef9b6b059cfc9ed9a6bc02178800c48:

This test needs to use SerialMesh since nodes get renumbered with ParallelMesh.

Refs #2105.
jwpeterson commented 10 years ago

In d8701482522c1a355a7d2206d9939ea0f3921f92:

Throw an error (in opt mode) if StripeMesh is used with ParallelMesh.

In dbg mode, StripeMesh will trip an assert in libmesh when used with
ParallelMesh.

Refs #2105.
jwpeterson commented 10 years ago

In aa1dbba1e827c7f70cc287237cac2a27ced1a658:

The nemesis_out_test can only be run on one processor if you
expect to find the file out.e.1.0, since the number of processors
is encoded into the name.

Refs #2105.
jwpeterson commented 10 years ago

In f3bb97b11b03728b325092e6725cec0628dd2e89:

Don't blindly pass the argument following --yaml/--dump to the parser,
check to see if it starts with a dash first.

Refs #2105.
jwpeterson commented 10 years ago

In 7db6e8a7ab3f4d981ef5dd9e51eef907c2363d99:

The meshes for these input files only have 1 element, so they don't
work if you try to run them in parallel with ParallelMesh.  Therefore,
restrict these tests to using SerialMesh.

Refs #2105.
jwpeterson commented 10 years ago

In cbdb37545e5e73a8b81bcb28ce9f833cd48b6b3c:

Make the reinitNodes() function a bit safer when run with ParallelMesh
by using the MeshBase::query_node_ptr() functionality.

Refs #2105.
jwpeterson commented 10 years ago

In 3efe967fed8cccb114b882507ce2ad1f13021592:

Make the ElementalVariableValue postprocessor safe to use with ParallelMesh.

The MeshBase::query_elem() function never asserts and returns NULL for
non-local elements.  It is the safe way to get Elem pointers whenever
ParallelMesh may be used.

Refs #2105.
jwpeterson commented 10 years ago

In a08ae9ba44977bdff9f4dc1932f4ac399bd76d57:

Restrict SolutionUserObject tests to run with SerialMesh.

The SolutionUserObject uses the copy_nodal_solution() capability of
the Exodus reader, and therefore won't work if the initial mesh has
been renumbered (it will be reunumbered if you are running with
ParallelMesh in parallel).  Hence, we restrict the relevant tests to
run with SerialMesh only.

Refs #2105.
jwpeterson commented 10 years ago

In 16ca8b8e8a5fc9f43c5a77ba754798fe18308caa:

Handle cases where PointLocator returns a NULL pointer due to ParallelMesh.

Refs #2105.
jwpeterson commented 10 years ago

In 6c98c25ae5a4677e6adc365411727c3d6f722fe7:

Restart from Exodus files relies on ExodusII_IO::copy_nodal_solution(),
which only works with SerialMesh.

Refs #2105.
jwpeterson commented 10 years ago

In [repository="" revision="20905" Don't access (20905]:


Be consistent with other string checks in the same file.

Refs #2105.
jwpeterson commented 10 years ago

In f3bd5b67d821c3e21445f8deccc38d48d7942ac2:

At Cody's request: printing what type of Mesh we are using.

Refs #2105.
jwpeterson commented 10 years ago

In e96a8eb0feb13524423db33033f20ca4adfa4e96:

Writing element data works with ParallelMesh if we use a MeshSerializer
object.

Refs #2122
Refs #2105
jwpeterson commented 10 years ago

In a1ef93a21b46e7dfb2e9fffd38c106024af07878:

This test calls ExodusII_IO::copy_elemental_solution() which only
works with SerialMesh.

Refs #2105
jwpeterson commented 10 years ago

In 43fc7f31c7c05e934acfbb8a4c1146c31f796cdf:

Adding errorIfParallelDistribution() function to MooseMesh.

This function can be used by clients of MooseMesh that are known not
to work with ParallelMesh to throw a uniformly-formatted error in
cases where they are incorrectly used by ParallelMesh.

Adding MooseMesh::errorIfParallelDistribution() to the
NodalVariableValue class; will be adding it to others over time.

Refs #2105.
jwpeterson commented 10 years ago

In 30f116c4c0b90cdf2ac90e0b6ab5e755d181d9f6:

Throwing errors in objects that don't work with ParallelMesh

* StripeMesh
* SideSetsFromPoints
* SideSetsFromNormals
* AddAllSideSetsByNormals
* TiledMesh
* ElementalVariableValue
* SolutionUserObject
* MultiAppNearestNodeTransfer
* MultiAppMeshFunctionTransfer
* MultiAppInterpolationTransfer
* MultiAppUserObjectTransfer

Refs #2105.

Adding distribution=serial to input several input files to avoid triggering the
new error message.
YaqiWang commented 10 years ago

MeshOutput for creating the exodus file needs to be parallelized. The current implementation, passing all data to the master processor is not good. It creates a memory usage spike and could causes jobs terminated abnormally. Because elements on the same block could be distributed among processors, we might need to creating new blocks for the original blocks such as proc0_0. I am sure we can merge those block later if it is desired. I donot know if we can have the parallel IO on exodus files, but even sequential output should not take too much time.

friedmud commented 10 years ago

Nope - this is what Nemesis is for. If you are using parallel mesh you should be outputting Nemesis format instead of Exodus.

We know that there are currently a few feature gaps in the Nemesis writing (like the ability to output elemental variables for instance). If you need that stuff make a ticket for those gaps.

YaqiWang commented 10 years ago

Does SolutionUserObject support Nemesis?