Cross Validation of Convection Parameterisations

[jatkinson1000]

As we develop/extract convection parameterisation NN (ConvParamNN) code from SAM and insert it into CAM we will be developing an interface layer that wraps around the NN to perform changes of variables etc (see #2).

To evaluate performance in CAM we want some kind of testing framework that compares its existing convection parameterisation (ConvParamCAM, also known as ZM from Zhang & McFarlane (1995)) to our new one.

This will involve a test rig that takes a set of inputs to the CAM parameterisation, and passes them to both ConvParamCAM and our wrapper layer to ConvParamNN.

Both approaches should run and return outputs, which we can then compare the results to evaluate performance and help indicate whether that the wrapper layer is processing CAM inputs appropriately.

It might also be nice if we can develop some standardised/unit tests for the wrapper layer performing physical variable transformation that are independent of the NN.

[paogorman]

Comparing to Zhang and McFarlane will be useful but not a very strong test because (1) we expect the new NN parameterization to behave differently to a conventional parameterization and (2) even two conventional parameterizations would not behave similarly for one input time (statistics accumulated over time would be similar)

Here is some brainstorming of additional tests that might be useful: 1) Make sure that the transformation (qv,qcc,qii,tabs)<->(q,t) returns the same values when we use it in the forward direction followed by the reverse direction

2) Implement the new parameterization with CAM type inputs (qv, qcc, qii, tabs) in SAM and see that it gives similar results to the original implementation.

3) Implement the new parameterization in SAM (as above) but now running SAM with a slightly different vertical grid z and/or a longer time step. This would test the interpolation and subcycling. Note the time step can't be much longer or SAM will go unstable.

4) For a few sample temperatures and values of q (some high q so there is cloud and some low q so there is no cloud), calculate the transformation (q,t)->(qv,qcc,qii, tabs) and compare the results with an independent calculation using equations in the appendix of the original SAM paper (https://doi.org/10.1175/1520-0469(2003)060<0607:CRMOTA>2.0.CO;2)

[jatkinson1000]

As discussed by Paul in comments above and in meetings, we do not want to conduct a direct comparison to the ZF parameterisation as we expect them to be different.

Whilst itr may be interesting to see how the NN parameterisation results compare to Zf on a benchmark, there is no need to cross-validate as is proposed by this issue.

Therefore I propose we close this at the next review meeting.

The comments from Paul about testing and integration should be transferred to:

• the Neural_Network_parameterisation repo where @raehik and @jatkinson1000 are testing the reintegration of the refactored NN with the SAM code.
• An appropriate issue about enforcing physical compatibility in variable changes relating to #18

as appropriate

[jatkinson1000]

As discussed, comments have been transferred elsewhere and this issue is now being closed.