Field diagnostic coarsening ratio requires the boxes to be coarsenable

[NeilZaim]

So if we want to have a coarsening ratio of n in one direction, the size in this direction of every box must be a multiple of n.

When we use max_grid_size and blocking_factor to do the domain decomposition, this can be satisfied by requiring that n divide the blocking factor in this direction. This is what is specified in the docs and we even have an assert for that.

Now, if we use warpx.numprocs for the domain decomposition, this is a bit more complicated and this requirement means that the grid size in one direction must be a multiple of numprocs*coarsening_ratio in this direction. This is clearly neither a big nor an urgent issue but this can be slightly inconvenient to have to recalculate the grid size every time we change e.g. the resolution for a convergence scan.

In the short term, I think it would be useful to update the docs and asserts to take into account to possibility that warpx.numprocs is used (I'll try to write a small PR for that later today).

In the longer term, it could be convenient to remove this constraint so that we could accept any value for the coarsening ratio. For instance, we could dump points that are located at geom.Domain().smallEnd() + integer multiples of the coarsening ratio, regardless of the domain decomposition. Would there be fundamental problems with doing that?

[ax3l]

Raised again today by @tmsclark, who wants to work on this.

attn @WeiqunZhang maybe you can give him some guidance? :)

[WeiqunZhang]

There are two issues. I suppose the first issue of the whole domain not being a multiple of the coarse ratio is not a big issue. But we do want to fix the second issue due to the size of individual boxes. For that, my suggestion is we create a temporary BoxArray that has nice sizes and use that for coarsening.

BoxArray ba(Geom(0).Domain());
ba.maxSize(nice_box_size);
MultiFab tmp(amrex::convert(ba,index_type), DistributionMapping(ba), ...);
tmp.ParallelCopy(...);