quokka-astro / quokka

Two-moment AMR radiation hydrodynamics (with self-gravity, particles, and chemistry) on CPUs/GPUs for astrophysics
https://quokka-astro.github.io/quokka/
MIT License
46 stars 12 forks source link

add MultiFabManager / ParticleContainerManager classes #748

Open BenWibking opened 1 month ago

BenWibking commented 1 month ago

Describe the proposal We often need to do the same set of operations over all MultiFabs and ParticleContainers, but C++ doesn't provide a capability to iterate over all of the member variables in a class, so we have to hard code them. However, can instead dynamically allocate MultiFabs and keep track of a list of all of the MultiFabs ourselves.

WarpX does this with its MultiFabRegister. This creates an object that behaves like a dictionary (e.g., in Python), where we can access (or create) the MultiFabs with a string, and also loop over all of them with an iterator.

We also need to do this with particle containers, since we may have many particle containers, and we don't want to hard code everywhere we need to redistribute particles, write them to disk, etc., for each particle container individually.

If we need to use the MultiFabs inside a amrex::ParallelFor kernel, we just get a reference to the MultiFab from the MultiFabManager while outside the ParallelFor, and from then on, use it as we normally would.

Describe alternatives you've considered Keep as-is. If we need to loop over all MultiFabs or ParticleContainers, we do it manually.

Additional context https://github.com/ECP-WarpX/WarpX/blob/fb9e7dd1a2946c1f293cd25f2061d78cf5d1d71f/Source/ablastr/fields/MultiFabRegister.H#L156

BenWibking commented 1 month ago

This will allow us to split state_old_cc_ and state_new_cc_ into separate hydro and radiation multifabs, which should have a small positive performance impact (since the radiation multifab does not need to have its ghost cells filled during the hydro update).

BenWibking commented 1 month ago

Also, this will allow us to split the state_old_fc_ and state_new_fc_ multifabs into divergence-free and non-divergence-free components. This is needed for AMR MHD, since we have to use a different AMR interpolater for the magnetic field and the face-centered velocity.

BenWibking commented 1 month ago

For reference, this is the WarpX PR for its equivalent MultiFabRegister class: https://github.com/ECP-WarpX/WarpX/pull/5230