MPIStateArray revamp

[simonbyrne]

Currently MPIStateArray is somewhere in between a "pure" array structure and a specialized backing datastore for values on a DiscontinuousSpectralElementGrid. It has a bunch of compromises and assumptions that makes it a bit cumbersome to use.

My current thinking is that we could define a Field object (perhaps there is a better name) that represents a set of numerical quantities that are defined over a spatial domain. In other words, this would be equivalent to a current MPIStateArray + Grid + Vars.

• Move some "grid-aware" functionality out of MPIStateArray (e.g. weights and the weighted norm/dot) into the Field object
• Interpolation, integration, gradients, etc. could be defined directly on the Field object, and not require any external DGModel object
  • e.g. interpolation could return another Field backed by a different Grid type (e.g. LatLongGrid or RegularCartesianGrid).
• For splitting schemes (e.g. Stormer-Verlet #645), the DG tendencies only require an subset of input and output variables (which can be different). This could be handled by passing in Fields with a subset of Vars variables.
• We could overload getproperty on the whole Field which could give you a subset of the vars e.g. a scalar or vector field.

I haven't touched on issues like communication or ghost points, and I know others have opinions, so please chime in with your wishlist below.

[blallen]

Need to think a bit more about it, but I love the idea of splitting out functionality into a Field. This is really the base object we are working with and MPIStateArray is just how the data for it is stored.

[jkozdon]

@lcw and I have been thinking some about this over the past few months.

It seem that we have too many ideas in MPIStateArrays and need to split some out.

It would also be nice to be semi-agnostic about data layout, e.g., do we want state to be fastest, which degree of freedom first (i, j, k), element order, etc.

We've been thinking that a series of wrapper array types could do this for us. Something along the lines of the following

GhostedArray:

• Array that has a real view and a ghost view

StructArray / VarsArray

• Allows Q.a returns a view into Q
• Q.a should return another VarsArray
• Q[I,j] should return a struct laid out like vars specifies. If the Vars only has one field, it should return whatever the type of the vars field is
• This is what lead to the imperfect modification of MPIStateArrays in PR #731
• I've played a bit around with this (d677ebd15b509a2e665be1c447e9992b96f3d934), but still needs a lot of work

WeightedArray

• Allows norm to be computed correctly with integration

MPICommArray

• Combine arbitrary sets of ghosted arrays (and views of ghosted arrays) for MPI communication
• norms should work
• reductions should be MPI aware as well. If you do a reduce along the vertical the result should be stored in parallel, reductions in the horizontal should result in all reduces

On top of all this

• Everything should be adapted correctly so that basic array operations work
• Which of these need to work on the GPU needs to be considered

[simonbyrne]

After discussion with @blallen, @sandreza and @leios:

1. Take #731 and make a new "Field" object basically consisting of

   struct Field{V<:Vars,A,G}
   array::A
   grid::G
   end

2. Figure out recursive Vars (I have some ideas)

3. Instead of passing state, aux, diffusive etc to every function, we pass a single NamedTuple containing the Vars objects: this will considerably simply the function signatures, especially now that #658 is merged.

4. Figure out a way to have a filteredstate inside the main DG loop, that can get passed as another element of the NamedTuple in 3 (related #775).

5. Be able to set boundary_state! once for all fluxes.

[simonbyrne]

Need to support broadcasting a horizontal-only field into 3D space. Ideally would capture the necessary restrictions in the grid object