search

idaholab / moose

Multiphysics Object Oriented Simulation Environment
https://www.mooseframework.org
GNU Lesser General Public License v2.1
1.75k stars 1.05k forks source link

Partitioner with weights according to DOFs on each block #13104

Open snschune opened 5 years ago

snschune commented 5 years ago

Reason

Better load balancing for problems with varying DOFs per block.

Design

Inherit from PetscPartitioner and override weight assignment.

Impact

New capability

GiudGiud commented 2 weeks ago

I think this could be done with a re-partitioner that runs later. The schedule of tasks prevents doing this directly right now

keeping track of this here

28793

lindsayad commented 1 week ago

I don't quite understand what this issue is after. What are we trying to weight for exactly?

GiudGiud commented 1 week ago

This issue is after weighting the elements with the number of DOFs. I think rn that isnt done in any of our custom moose ones, but might be doable with the petsc one

lindsayad commented 1 week ago

The most difficult part of this is that the partitioning would have to be delayed until at least some setup has been done in the DofMap/DofObjects. Typically we have a partitioning and then we distribute the dofs according the pre-established partitioning. We would have to setup the number of dofs on each DofObject without actually doing any global dof numbering, and then performing the partitioning. And then go and do the global dof numbering. @roystgnr do you have thoughts on this?

GiudGiud commented 1 week ago

Yes that s the problem that blocked Sebastian s PR.

However I think we could re partition with another partitioner much later to achieve that. We could keep our setup much of the same, then re-partition. Do we need to renumber when we repartition? I imagine the objects we create might not like late renumbering?

roystgnr commented 1 week ago

Renumbering before partitioning is slower, but only asymptotically slower if you're on a DistributedMesh and are thereby also forgoing the ability to distribute before renumbering.

Counting DoFs before (well, during) partitioning would be possible ... and would probably be the easiest thing to quickly implement efficiently?

But weighted repartitioning from an initial DoF-numbered partitioning might be the least total work, since that would work both for initial partitioning and for adaptive repartitioning after mesh changes later.

Arbitrary weighting is only supported in Moose PetscExternalPartitioner, but the libMesh interfaces to Metis and Parmetis have their own built-in weighting ("number of nodes per element"), and it would actually be pretty easy to replace that with "number of nodes per element if we see no DoF indices, number of DoF indices per element (including those on its nodes) if we do". Actually, that sounds so sensible now that I spell it out that I might want to do it regardless of whether we do something at the MOOSE level too.