WeiqunZhang
commented
2 years ago The refinement creates enough boxes for all the nodes at the given level, but some processors have 0 boxes owned and some have 2.
This is not supposed to happen. Could you try the latest AMReX? Is it possible to provide a reproducer?
Hi, I'm writing an entropic lattice Boltzmann method (ELBM) flow solver in Fortran and using Amrex (currently 20.09 iirc) to the grid refinement and bookkeeping for it. The issue I'm having stems from how the distribution mapping affects the loads on a given levels. The nature of ELBM means needing to loop through an entire level and do work on it before going to the next. Unfortunately, as I'm refining the grid, the grids don't seem to be balanced.
I've been working on a coarse time function, run the flow on coarser meshes then refine to a desired baseline once enough time has passed and continue as normal with AMR functionality. The problem occurs when running on threaded nodes. This is all done on my workstation atm which has a 6 core, 12 thread processor. The refinement creates enough boxes for all the nodes at the given level, but some processors have 0 boxes owned and some have 2. At the next refinement level, there are more boxes, but some of the processors own 3 and most only own 1.
I'm not sure how to balance the load amongst the processors to make sure every node gets at least 1 box for calculation on a level. I think the problem will lessen as I move from testing to larger cases and processor counts, but it raises a few questions. Any idea on the cause of this? As I'm using Fortran, my access to the Distribution Mapping functions is limited, Is there a way to ask Amrex to redistribute the load?
I wasn't sure if it was a bug or some effect of my system (so please move the thread as needed).