Add more support for N dimensions

[AlexanderSinn]

Some codes, such as HiPACE++, use a different number of dimensions in different parts of the code. Currently, AMREX_SPACEDIM has to be chosen as the maximum number of dimensions used, with the other places in the code not utilizing the extra dimensions by assigning a size of 1 to them. This can be very confusing, and in the case of ParallelFor and Array4 can lead to a slight performance loss. Additionally, it might be useful to have more than 3 dimensions sometimes. For example, a 3D loop with tiling on GPU could be implemented with a 6D ParallelFor.

This can be achieved by replacing the AMREX_SPACEDIM macro with a template parameter that specifies the number of dimensions.

I would volunteer to start working on this by converting IntVect to IntVectND<> and adding using IntVect = IntVectND<AMREX_SPACEDIM>;. Currently IntVect relies hevily on AMREX_D_TERM. This can be replaced by sfinae, if constexpr or for loops that will usually get unrolled since the bounds are known at compile time.

Later on, the same can be done for other types such as RealVect, Box, Geometry, Array4, as well as ParallelFor.

Even if not everything will be converted (BoxArray, MultiFab might require too much effort), just having N-dimensional indexing would be very useful.

[WeiqunZhang]

Maybe we should have a meeting discussing this.

[AlexanderSinn]

Good point, maybe at the next WarpX+AMReX development meeting?

[AlexanderSinn]

Now that ParallelFor is N dimensional, I think a base level of functionality for N dimensions is provided. Below, I listed some additional classes that could be converted to N dimensions similarly to IntVect/Box/ParallelFor. I have not started on any of these, so anybody can feel free to contribute.

• [x] IntVect #3969
• [x] IndexType #3988
• [x] Box #4016
• [x] ParallelFor #4052
• [x] ParallelFor CTO #4109
• [ ] Reduce
• [ ] Array4
• [ ] RealVect
• [ ] RealBox
• [ ] Geometry