Add temporary buffers support for atmosphere processes

[bartgol]

The AD won't likely use the WorkspaceManager from ekat. It will likely have something simple, like the FunctorsBuffersManager in Hommexx (which simply allocates a 1d buffer that accommodates all needs). Each atm process should then use the memory pointer provided by the AD. That's when the WSM from ekat will likely be used, i.e., created inside each atm processes, with memory provided from the AD.

Action items:

• add interfaces to atm process for getting the required memory size, and for setting a (device) memory ptr.
• make the AD query atm procs for needed temp memory, allocate memory, and finally set memory inside the atm procs.

[tcclevenger]

@bartgol Is the plan to no longer use WSM inside the main functions (e.g. shoc_main()), but instead create and pass through a new structure which manages AD memory (the FunctorsBuffersManager-like structure)? Or is it just to use aFunctorsBuffersManager-like structure inside the AD for the non-FM variables passed to the main function, with shoc impl being completely agnostic to its existence?

I think this task will be related to some temp variable work I need to do in shoc (making tracer transpose a workspace variable), so I may take this issue.

[bartgol]

I don't really care how we achieve the result, but the logic should be roughly this: first the AD queries all atm procs for their temporary needs. Then, the AD allocates a single buffer that accommodates all needs, and pass the buffer back to all atm procs.

The second step could be achieved by having the AD store a single WSM (instantiated to accommodate the largest need among all atm procs), and provide it to all atm procs. However, different Atm Procs might use different Team policies, or even need temps on different grids. Ideally, e.g., I'd like Homme's FunctorsBufferManager to use the scratch mem provided by the AD.

We can discuss more via slack, and report here our conclusions, if you want, but my guess is that the AD should gather raw-memory needs from all atm procs, and provide back a raw-memory buffer. Each atm proc is then free to use this in whatever way they want. If we go this route, we need to provide a ctor of WSM that also takes in the raw mem buffer (without allocating it). It would be "risky", since WSM would have no way to check that the provided mem buffer is long enough. But it would be marked as a power-user ctor, with all the documentation needed to be aware of risks/assumptions.