ForestClaw / forestclaw

Quadtree/octree adaptive PDE solver based based on p4est.
http://www.forestclaw.org
BSD 2-Clause "Simplified" License
57 stars 21 forks source link

Partitioning strategies for patch data #335

Closed hannesbrandt closed 2 weeks ago

hannesbrandt commented 3 weeks ago

This PR adds a new interface for partitioning patch data in different modes, with one of them being already implemented.

We introduce a new workflow for partitioning patch data:

  1. The patch data is packed into send buffers by calling fclaw2d_domain_iterate_pack. The function receives a user-defined fclaw2d_pack_callback_t, which packs the data of a patch into a provided storage location. Then the function starts sending the packed data according to the new partition boundaries.
  2. While waiting for the communication to complete, the data of patches which stay local can be transferred between the old and the new domain with fclaw2d_domain_iterate_transfer. The function uses a fclaw2d_transfer_callback_t, which may be a simple copy or reassignment of the patch data. fclaw2d_domain_iterate_transfer may be called before iterate_pack or after iterate_unpack instead.
  3. Every call to fclaw2d_domain_iterate_pack has to be succeeded by a call to fclaw2d_domain_iterate_unpack. This function unpacks the repartitioned patch data it received by a user-defined fclaw2d_unpack_callback_t.
  4. Finally, a call to fclaw2d_domain_partition_free frees all partition-specific context data.

The workflow has to be called after fclaw2d_domain_partition, which repartitions the patches without the patch data and updates the partition boundaries as required. For now we keep the old workflow of using assign_before_partition, retrieve_after_partition and iterate_partitioned. It may be useful for writing a comparison paper on different partitioning modes.

The implementation currently comes with a single mode that packs all patches into the send buffer, even if they are not sent after all. For future additional modes, we propose to add two partitioning options, skip_local and skip_refine, which can be combined arbitrarily:

Does this interface with the corresponding options seem convenient to you? We would appreciate your feedback/ideas on this topic!

scottaiton commented 3 weeks ago

The interface and options look good! I don't have any requests for changes.

hannesbrandt commented 2 weeks ago

Great, then I will turn this into a normal PR, mainly for the interface. The skip_local and skip_refine options will follow in future PRs.