fdrmrc / Polydeal

C++ implementation of Polygonal Discontinuous Galerkin method within the deal.II Finite Element library.
https://fdrmrc.github.io/Polydeal/
Other
0 stars 0 forks source link

Agglomerate distributed mesh #86

Closed fdrmrc closed 9 months ago

fdrmrc commented 9 months ago

To be merged after #85

This PR enables the partition of locally owned regions of a distributed mesh by using the function PolyUtils::partition_locally_owned_regions().

What this function is doing is essentially the same as SparsityTools::partition(), but local to each process. The idea is to call METIS within each partition, in order to generate a given number of agglomerates in each locally owned region.

Doing this requires some additional steps due to the fact that partitions generated by p4est can be discontinuous, see partitions $1$ and $4$ in the next picture.

image

This implies that METIS will throw because the local graph to each process is not continuous. The workaround for this is to use a parallel::fullydistributed::Triangulation and repartition this triangulation before calling partition_locally_owned_regions, in order to have continuous partitions within each process.

On that hypercube with a strided partitioner , asking for 10 agglomerates in each partition with a triangulation distributed among $3$ processors amounts to call

PolyUtils::partition_locally_owned_regions(10,
                                             tria_pft,
                                             SparsityTools::Partitioner::metis);

and generates the following local agglomerates in each one of the $3$ MPI ranks:

Process 0 Process 1 Process 2
image image image

Do you see any potential issue @luca-heltai ?

fdrmrc commented 9 months ago

I've added a test which does sanity checks using also a fully distributed tria and exploits the partitioning strategy described above. The relevant changes are:

https://github.com/fdrmrc/Polydeal/blob/fa82876ffc2dabe747ee07dd981fb5d6f7e78005/test/polydeal/fully_distributed_poisson_sanity_check_01.cc#L144-L172

fdrmrc commented 9 months ago

I quickly tested with the Rtree and the sanity check runs smoothly. Here the agglomerates in each partition:

Process 0 Process 1 Process 2
image image image
fdrmrc commented 9 months ago

Finally, I've tested this with a fully distributed tria constructed out of an external mesh (see here https://dealii.org/developer/doxygen/deal.II/namespaceTriangulationDescription_1_1Utilities.html#a0411d757cd85a77d25bbb9303af93de7)

This gives automatically continuous partitions (it's using METIS) and hence no repartitioning policy is required. Moreover, the generated agglomerates look nicer. (As before, with $3$ processes and 10 agglomerates per process)

Process 0 Process 1 Process 2
image image image
luca-heltai commented 9 months ago

Nice. Very nice. (Y)

fdrmrc commented 9 months ago

I'm merging this (I have already tested it on some 3D mesh)