Create a way to store persistent non-local data

[charleskawczynski]

Description

Per a discussion with @kmdeck, we need a way to compute non-local information (e.g., data in boundary_state!) and make this information available in flux_first_order!,flux_second_order!, and source!. For example, we would like to have something like

model = LandModel(;bc_param = [2.0])

and update this value in, for example, update_auxiliary_state!(::LandModel, ...). The main issue with this is that ordinary arrays cannot reside on the GPU. Using an MArray will not work because it cannot persist beyond the kernel calls. One hack option would be to create an entirely new instance of the balance law inside update_auxiliary_state!(::LandModel, ...). Something like

# In driver:
model = LandModel(;bc_param = SVector(1))

# in LandModel.jl
function update_auxiliary_state!(
    dg::DGModel,
    m::LandModel,
    Q::MPIStateArray,
    t::Real,
    elems::UnitRange,
)

m = mutate_model(m; bc_param = SVector(2))

end

This assumes that the parent of m can mutate m (which should be doable). The downside of this is frequent reallocation of land::LandModel, which could be problematic if we need to eventually store datasets in land::LandModel.

Setfield.jl might be useful here.

[kmdeck]

We'll definitely have data sets at some point, and I know @Yujie-W has been working on how to access/deal with them for other components of the land model. How does the canopy model access needed data sets? We can also discuss via slack or zoom!

[Yujie-W]

Currently, there are tons of local data stored in the sttructure of Land model using ordinary arrays. So the implementation of the Land model we have now into ClimateMachine.jl can be tough. Yet, it is possible to store these data outside the land model into a mutable cache stucture (if GPU can visit and change this cache structure).