[recipe testing strategy] Test combinations of preprocessor functions

[remi-kazeroni]

Is your feature request related to a problem? Please describe. During the meeting on the recipe testing strategy #2259 on October 14, 2021, we discussed the different options to test recipes on a daily basis. The idea is to check whether changes in the ESMValCore affect recipes and spot that earlier on than shortly before a release. That testing needs to be done in an automated way. The following points were discussed:

• Could we define a couple of recipes that would test all sorts of sequences of preprocessor functions at once and possibly all the preprocessor settings? Most of the current example recipes only test one preprocessor. This would help checking how changes in preprocessors affect recipes and also help capturing changes in preprocessor arguments.
• What kind of computational resources would be needed for that? Are the resources available with the ESMValBot enough?
• Such a testing would also help to better gauge the impact of backwards incompatible changes on the Tool. The handling of such changes (communication by email to users?) and their frequency may deserve further discussion.
• It is generally advised to develop recipes and diagnostics against the stable version of the Core rather than the latest development version. There may be cases where newly added features or fixes to the Core are required. This triggered the question of whether the release frequency of the Core should be increased.

Feel free to edit this summary or to comment in this issue.

[bouweandela]

As I understood it, the main result from this discussion was that we would implement tests that test commonly used sequences of preprocessors, e.g. first compute some temporal statistics and then multimodel statistics, not entire recipes. These tests could be quite fast if we use small test cubes, so this could be run as part of the normal unit tests.