search

idaholab / moose

Multiphysics Object Oriented Simulation Environment
https://www.mooseframework.org
GNU Lesser General Public License v2.1
1.71k stars 1.04k forks source link

Force Repartition #14324

Open friedmud opened 4 years ago

friedmud commented 4 years ago

Reason

After mesh generators run you might want to repartition the mesh. Right now - that won't happen.

Design

Add a parameter to the Mesh block to force_repartition

Impact

None - new option

lindsayad commented 4 years ago

@friedmud @permcody @fdkong Do we ever repartition say during large displacement problems? Looking at the code, I think no?

fdkong commented 4 years ago

@friedmud @permcody @fdkong Do we ever repartition say during large displacement problems? Looking at the code, I think no?

Whenever we call prepare for use, a partition could be involved.

Most of the time, we partition only once.

lindsayad commented 4 years ago

Yea I don't think we ever call prepare_for_use after the initial setup

fdkong commented 4 years ago

Yea I don't think we ever call prepare_for_use after the initial setup

For regular simulation, it is right. For mesh adaptivity, we do call prepare_for_use when the mesh is changed.

Repartition is expensive because you have to recreate matrix and vectors, etc.

What issues did you hit? Why you were asking?

lindsayad commented 4 years ago

For regular simulation, it is right. For mesh adaptivity, we do call prepare_for_use when the mesh is changed.

Sorry yea, of course you are right.

Repartition is expensive because you have to recreate matrix and vectors, etc.

What issues did you hit? Why you were asking?

Just thinking about consequences if your slave node starts projecting onto master faces with a different processor ID. Seems like it probably incurs a fair amount of communication

lindsayad commented 4 years ago

It's probably a CPU vs memory trade-off. If we repartitioned more frequently, we could probably use less ghost elements in geometric search problems. This isn't really relevant for ReplicatedMesh. But @acasagran has talked to me about a problem he ran once where he had to use less processes per node because he was running out of memory. It seems like if he could use DistributedMesh with a reasonable amount of ghosting, then he could be free of this issue.