Need discrete operator functions

[charleskawczynski]

We may need to iterate on the form of this to-do item, and we've discussed this but I'd like to just record the need here and discuss some pseudo-code examples.

The EDMF scheme (and likely other parameterization schemes) will need discrete operators (grad, div, etc.), provided by the dycore. Here's a pseudo-code example that I'm imagining:

Adds the derivative of `state_vector[f_sym]` with respect to
the direction, denoted by `dim`, on mesh element `elem`, and
stores the result in `state_vector[∇_sym]`.

compute_∇_local!(state_vector::StateVector,
                  ∇_sym::Symbol,
                  f_sym::Symbol,
                  mesh::Mesh,
                  dim::Int,
                  ::DerivativeType,
                  elem::Element
                  ) where Element where DerivativeType

Example:

vars = (:u, :v, :w, :dudx)
state_vector = StateVector(mesh, vars)

for e in over_elems(mesh)

  compute_∇_local!(state_vector,
                    :dudx,
                    :u,
                    mesh,
                    1,
                    VolumeDerivative(), # may also need FluxDerivative() or something
                    e
                    )

end

[jkozdon]

For something like this to work efficiently, I think that we want to think of the equation as a system of PDES and not a scalar PDE (since we will want to limit how often we loop over the arrays). Also loop over elements will (likely) be implicit on the GPU.

We may want to write a fully functional 1-D solver DG for a representative system of equations before figuring out what the abstraction should be. I fear coming up with an abstraction that is not as fully featured as we want/need.

A good start place for is the 1-D codes in Canary. Once we have a fully feature GPUified version we can think about abstractions.

[charleskawczynski]

Ok, well, #70 applies to Canary too, so I'll wait until the new mesh-encapsulated branch is merged. In the meantime, I've started writing some pseudo-code so that we can try to identify a minimum number of mesh-element traversals we can get away with.

[charleskawczynski]

Is this still an issue? I think #158 partially addresses this.

[jkozdon]

I'd say close for now. Hopefully #158 and following related PRs will address 99% of this need.