search

amanzi / ats

Advanced Terrestrial Simulator (ATS) development
Other
47 stars 35 forks source link

Logical mesh with parallel computing #269

Closed levuvietphong closed 2 months ago

levuvietphong commented 3 months ago

Problem:

The MeshFactory::createLogical function currently prevents parallel execution by enforcing AMANZI_ASSERT(comm_->NumProc() == 1);. This limitation restricts the creation of logical meshes to a single processor, which hinders scalability.

Proposed Solution:

The implementation should be updated to allow different processors to create logical meshes within their respective owned domains, enabling parallelism.

Example Test Case:

In the 09_multiscale_models/stream_hyporheic_conservative_tracer_reach test, the domain can be split across 2 (or more) processors as follows:

Since the subgrids are fully independent, this setup is ideal for parallel computing. The MeshFactory::createLogical function could then be called separately by P1 and P2 for their respective subdomains.

logicalmesh_parallel

ecoon commented 3 months ago

Note that this should work currently, the key is that the Domain mesh is already parallel (it is not a logical mesh, but an extracted, generated, or from file mesh) while the subgrid meshes themselves are each both logical and serial (they only live on MPI_COMM_SELF). The picture is a bit misleading, however -- each Domain1 and Domain2 own some set of cells, and 5 & 6 are owned by only one of Domain1 or Domain 2 (likely Domain1 owns 5 and Domain2 owns 6). Then the subgrid mesh associated with Cell 5 is only constructed, on MPI_COMM_SELF, by rank 0 (Domain1) in MPI_COMM_WORLD, while the subgrid mesh associated with Cell 6 is only constructed, on MPI_COMM_SELF, by rank 1 (Domain2) in MPI_COMM_WORLD.

levuvietphong commented 2 months ago

Closed as this has already been implemented. Regression tests on parallel runs were failed due to the use of a generated mesh for the stream domain, but new tests on the extracted mesh passed successfully.