Add regression test to benchcab

[ccarouge]

Regression testing is important to ensure we don't inadvertently break a functionality. For the moment, all analyses are done in me.org. The analysis script performs a scientific evaluation of the control and development outputs. Although it will clearly show the outputs are identical, it is more complicated and heavier to use than a simple regression test.

We should implement to run a bitwise comparison of the outputs after running the tasks. It can be done easily with cdo diff.

For optimisation, is it worth coding this so that the regression test for each experiment runs as soon as the outputs from both branches are written out? Or is it good enough to run the regression test once all the output is created? Considering we can run the regression tests in parallel at the end, it probably makes little difference to run all of them once all the outputs are created.

[SeanBryan51]

@ccarouge should we implement the bitwise comparison as a separate job? That is, have another subcommand: benchcab fluxnet-run-regression-test which by default submits a PBS job that does solely the regression test (or the regression test can be run on the login node with the --no-submit option). Alternatively, we could lump in the comparison step in the same PBS job that runs CABLE.

Pros/cons to using a separate job for the comparison:

• pro: if we use a separate job, the job would have to wait on the previous job (that produces CABLE output) before it can be executed. The qsub command supports the ability for jobs to wait on other jobs through the -W depend=... option (see the nci user guide or man page for qsub).
• pro: the comparison step is likely to have different memory and CPU requirements than what is required for running CABLE tasks, so it makes sense to have separate jobs.
• pro: this would allow users to run the comparison step and or the "running CABLE" step in isolation
• pro: when uploading the analysis to me.org becomes an automated step, we should ideally be using a copyq job to upload model outputs from Gadi. This will require its own PBS job that is dependent on the job that runs CABLE tasks.
• con: this would introduce extra logic in the code where we check that the "running CABLE" step is currently on the job queue / running. We can do this by:
  • reading from PBS environment variables to get the job id, and then running qstat <job_id>
  • storing the job id somewhere when submitting the job, and then running qstat <job_id>
  • If the "running CABLE" step is not completed, then we supply the -W depends=... argument to qsub, else we run qsub as per usual.
• con: it is easier to add an extra line in the "running CABLE" job script... but this is a bit of hacky solution.

What do you think?