search

fmihpc / vlasiator

Vlasiator - ten letters you can count on
https://www.helsinki.fi/en/researchgroups/vlasiator
Other
47 stars 38 forks source link

Ghost translation #1006

Open markusbattarbee opened 2 months ago

markusbattarbee commented 2 months ago

Replaces #609 Incorporates #1000

This PR introduces ghost translation (previously known as local translation). Allows for running all Vlasov translation with coalesced MPI just before the actual call, reducing the overhead related to 3D imbalances. This comes at the cost of having to actually compute translation for a boundary of ghost cells. This also requries a grid neighborhood of VLASOV_STENCIL_WIDTH+1 which does incur some overhead.

Turn on ghost translation with vlasovsolver.GhostTranslate set to true. This current form of the PR forces this on so we get CI with it active, but it should probably be turned off as default.

The correctness of this method was validated with local tests but this PR should re-confirm it with CI. If so preferred, a smaller ghost neighbourhood can be translated instead of all required, resulting in some diffs at process boundaries, but less of an overhead. This might be useful for example for early flushing of a simulation. The size of the stencil to include in ghost translation can be set withvlasovsolver.GhostTranslateExtent. The parameter has a default of 0 which is automatically replaced with VLASOV_STENCIL_WIDTH+1. Values greater than that are capped. Thus, possible values are between 1 and VLASOV_STENCIL_WIDTH+1.

markusbattarbee commented 2 months ago

WIP (e.g. the default handling breaks the ionosphere tests, I see) and evaluating performance bottlenecks.